Skip to main content
added 316 characters in body
Source Link
Teun Vink
  • 17.4k
  • 6
  • 45
  • 70

snmp { health-monitor { interval 30; rising-threshold 60; falling-threshold 50; } }

event-options { policy MONITOR-CPU { events snmpd_health_mon_thresh_cross; attributes-match { snmpd_health_mon_thresh_cross.event-name matches "Health Monitor.+CPU.+rising"; } then { execute-commands { commands { "show system processes extensive"; } output-filename cpu-processes; destination local-flash; output-format text; } }
} destinations { local-flash { archive-sites { /var/tmp; } } } }

snmp {
    health-monitor {
        interval 30;
        rising-threshold 60;
        falling-threshold 50;
    }
}
 
event-options {
    policy MONITOR-CPU {
        events snmpd_health_mon_thresh_cross;
        attributes-match {
            snmpd_health_mon_thresh_cross.event-name matches "Health Monitor.+CPU.+rising";
        }
        then {
            execute-commands {
                commands {
                    "show system processes extensive";
                }
                output-filename cpu-processes;
                destination local-flash;
                output-format text;
            }
        }                               
    }
    destinations {
        local-flash {
            archive-sites {
                /var/tmp;
            }
        }
    }
}
 

DISPLAY SET OUTPUT> set snmp health-monitor interval 30 set snmp health-monitor rising-threshold 60 set snmp health-monitor falling-threshold 50 set event-options policy MONITOR-CPU events snmpd_health_mon_thresh_cross set event-options policy MONITOR-CPU attributes-match snmpd_health_mon_thresh_cross.event-name matches "Health Monitor.+CPU.+rising" set event-options policy MONITOR-CPU then execute-commands commands "show system processes extensive" set event-options policy MONITOR-CPU then execute-commands output-filename cpu-processes set event-options policy MONITOR-CPU then execute-commands destination local-flash set event-options policy MONITOR-CPU then execute-commands output-format text set event-options destinations local-flash archive-sites /var/tmp

set snmp health-monitor interval 30
set snmp health-monitor rising-threshold 60
set snmp health-monitor falling-threshold 50
set event-options policy MONITOR-CPU events snmpd_health_mon_thresh_cross
set event-options policy MONITOR-CPU attributes-match snmpd_health_mon_thresh_cross.event-name matches "Health Monitor.+CPU.+rising"
set event-options policy MONITOR-CPU then execute-commands commands "show system processes extensive"
set event-options policy MONITOR-CPU then execute-commands output-filename cpu-processes
set event-options policy MONITOR-CPU then execute-commands destination local-flash
set event-options policy MONITOR-CPU then execute-commands output-format text
set event-options destinations local-flash archive-sites /var/tmp

show ddos-protection protocols statistics brief show ddos-protection statistics show ddos-protection version

show ddos-protection protocols statistics brief
show ddos-protection statistics
show ddos-protection version
 

Then depending what you see it can be narrowed down for example: show ddos-protection protocols ttl statistics show ddos-protection protocols ttl violations show ddos-protection protocols ttl flow-detection detail /this cm needs prior config/

show ddos-protection protocols ttl statistics
show ddos-protection protocols ttl violations
show ddos-protection protocols ttl flow-detection detail  */*this cm needs prior config*/*

set cli timestamp show chassis routing-engine (multiple snapshots, atleast 5) show system processes extensive (multiple snapshots atleast 5) show system users show system connections show system statistics

set cli timestamp
show chassis routing-engine (multiple snapshots, atleast 5)
show system processes extensive (multiple snapshots atleast 5)
show system users
show system connections
show system statistics

Turn on task accounting and collect the task accounting detail output (three times with a gap of 30 seconds). Don't forget to turn it off after finished.

set task accounting on show task accounting detail set task accounting off

show task memory detail show task memeory summary show task io show task history show task statistics show task job show task jobs show krt queue show krt state

set task accounting on 
show task accounting detail
set task accounting off

show task memory detail
show task memeory summary
show task io
show task history
show task statistics
show task job
show task jobs
show krt queue
show krt state

user@router# show routing-options traceoptions { file routing-trace size 10m files 20 world-readable; flag task; flag state; flag timer; }

user@router# show routing-options 
traceoptions { 
file routing-trace size 10m files 20 world-readable; 
flag task; 
flag state; 
flag timer; 
}

snmp { health-monitor { interval 30; rising-threshold 60; falling-threshold 50; } }

event-options { policy MONITOR-CPU { events snmpd_health_mon_thresh_cross; attributes-match { snmpd_health_mon_thresh_cross.event-name matches "Health Monitor.+CPU.+rising"; } then { execute-commands { commands { "show system processes extensive"; } output-filename cpu-processes; destination local-flash; output-format text; } }
} destinations { local-flash { archive-sites { /var/tmp; } } } }

DISPLAY SET OUTPUT> set snmp health-monitor interval 30 set snmp health-monitor rising-threshold 60 set snmp health-monitor falling-threshold 50 set event-options policy MONITOR-CPU events snmpd_health_mon_thresh_cross set event-options policy MONITOR-CPU attributes-match snmpd_health_mon_thresh_cross.event-name matches "Health Monitor.+CPU.+rising" set event-options policy MONITOR-CPU then execute-commands commands "show system processes extensive" set event-options policy MONITOR-CPU then execute-commands output-filename cpu-processes set event-options policy MONITOR-CPU then execute-commands destination local-flash set event-options policy MONITOR-CPU then execute-commands output-format text set event-options destinations local-flash archive-sites /var/tmp

show ddos-protection protocols statistics brief show ddos-protection statistics show ddos-protection version

Then depending what you see it can be narrowed down for example: show ddos-protection protocols ttl statistics show ddos-protection protocols ttl violations show ddos-protection protocols ttl flow-detection detail /this cm needs prior config/

set cli timestamp show chassis routing-engine (multiple snapshots, atleast 5) show system processes extensive (multiple snapshots atleast 5) show system users show system connections show system statistics

Turn on task accounting and collect the task accounting detail output (three times with a gap of 30 seconds). Don't forget to turn it off after finished.

set task accounting on show task accounting detail set task accounting off

show task memory detail show task memeory summary show task io show task history show task statistics show task job show task jobs show krt queue show krt state

user@router# show routing-options traceoptions { file routing-trace size 10m files 20 world-readable; flag task; flag state; flag timer; }

snmp {
    health-monitor {
        interval 30;
        rising-threshold 60;
        falling-threshold 50;
    }
}
 
event-options {
    policy MONITOR-CPU {
        events snmpd_health_mon_thresh_cross;
        attributes-match {
            snmpd_health_mon_thresh_cross.event-name matches "Health Monitor.+CPU.+rising";
        }
        then {
            execute-commands {
                commands {
                    "show system processes extensive";
                }
                output-filename cpu-processes;
                destination local-flash;
                output-format text;
            }
        }                               
    }
    destinations {
        local-flash {
            archive-sites {
                /var/tmp;
            }
        }
    }
}
 

DISPLAY SET OUTPUT>

set snmp health-monitor interval 30
set snmp health-monitor rising-threshold 60
set snmp health-monitor falling-threshold 50
set event-options policy MONITOR-CPU events snmpd_health_mon_thresh_cross
set event-options policy MONITOR-CPU attributes-match snmpd_health_mon_thresh_cross.event-name matches "Health Monitor.+CPU.+rising"
set event-options policy MONITOR-CPU then execute-commands commands "show system processes extensive"
set event-options policy MONITOR-CPU then execute-commands output-filename cpu-processes
set event-options policy MONITOR-CPU then execute-commands destination local-flash
set event-options policy MONITOR-CPU then execute-commands output-format text
set event-options destinations local-flash archive-sites /var/tmp
show ddos-protection protocols statistics brief
show ddos-protection statistics
show ddos-protection version
 

Then depending what you see it can be narrowed down for example:

show ddos-protection protocols ttl statistics
show ddos-protection protocols ttl violations
show ddos-protection protocols ttl flow-detection detail  */*this cm needs prior config*/*
set cli timestamp
show chassis routing-engine (multiple snapshots, atleast 5)
show system processes extensive (multiple snapshots atleast 5)
show system users
show system connections
show system statistics

Turn on task accounting and collect the task accounting detail output (three times with a gap of 30 seconds). Don't forget to turn it off after finished.

set task accounting on 
show task accounting detail
set task accounting off

show task memory detail
show task memeory summary
show task io
show task history
show task statistics
show task job
show task jobs
show krt queue
show krt state
user@router# show routing-options 
traceoptions { 
file routing-trace size 10m files 20 world-readable; 
flag task; 
flag state; 
flag timer; 
}
Source Link
DRP
  • 1.5k
  • 1
  • 8
  • 17

I know this thread is old but for sake of completeness:

If the high cpu occurs randomly and you are not able to determine the process causing this we can create the script below.

With this script we are going to capture the process extensive when a process raises more than the normal or expected threshold, this should not disrupt any traffic but a MW is still recommended. However i see you have narrowed it to be RPD.

snmp { health-monitor { interval 30; rising-threshold 60; falling-threshold 50; } }

event-options { policy MONITOR-CPU { events snmpd_health_mon_thresh_cross; attributes-match { snmpd_health_mon_thresh_cross.event-name matches "Health Monitor.+CPU.+rising"; } then { execute-commands { commands { "show system processes extensive"; } output-filename cpu-processes; destination local-flash; output-format text; } }
} destinations { local-flash { archive-sites { /var/tmp; } } } }

DISPLAY SET OUTPUT> set snmp health-monitor interval 30 set snmp health-monitor rising-threshold 60 set snmp health-monitor falling-threshold 50 set event-options policy MONITOR-CPU events snmpd_health_mon_thresh_cross set event-options policy MONITOR-CPU attributes-match snmpd_health_mon_thresh_cross.event-name matches "Health Monitor.+CPU.+rising" set event-options policy MONITOR-CPU then execute-commands commands "show system processes extensive" set event-options policy MONITOR-CPU then execute-commands output-filename cpu-processes set event-options policy MONITOR-CPU then execute-commands destination local-flash set event-options policy MONITOR-CPU then execute-commands output-format text set event-options destinations local-flash archive-sites /var/tmp

Also have you checked if any ddos messages have been reported? You could run the following commands:

show ddos-protection protocols statistics brief show ddos-protection statistics show ddos-protection version

Then depending what you see it can be narrowed down for example: show ddos-protection protocols ttl statistics show ddos-protection protocols ttl violations show ddos-protection protocols ttl flow-detection detail /this cm needs prior config/

Juniper also has a collection list for this type of issues under KB22637

High CPU CLI Commands

set cli timestamp show chassis routing-engine (multiple snapshots, atleast 5) show system processes extensive (multiple snapshots atleast 5) show system users show system connections show system statistics

Turn on task accounting and collect the task accounting detail output (three times with a gap of 30 seconds). Don't forget to turn it off after finished.

set task accounting on show task accounting detail set task accounting off

show task memory detail show task memeory summary show task io show task history show task statistics show task job show task jobs show krt queue show krt state

Logs

Archive /var/log as specified in Step 1 above Traceoptions

user@router# show routing-options traceoptions { file routing-trace size 10m files 20 world-readable; flag task; flag state; flag timer; }

Also if you are running an old version which made be prone for bugs, you might want to check the life support of the code:

http://www.juniper.net/support/eol/junos.html

Another point to mention which could be a vector attack is not having protected your RE from unwanted exception traffic. Make sure you have a firewall filter under the loopback.

I have seen in the past scripts on the router causing high cpu not sure if rpd came into my view, but this is something you might not want to overlook.

If you see in the logs many hits with RPD_MPLS_PATH_BANDWIDTH_CHANGE you might be using a very aggressive adjust-interval

Check any drops on "show system queue: this is the kernel queue, some clue may appear.