Cause of high CPU load on Juniper peering router's routing engine

Question

Recently the routing engine CPU utilization on two of our Juniper peering routers increased from ~10-20% average load to 80+%. I'm trying to figure out what's causing this (and how to get this high load back down).

Some info on the routers: both run the same JunOS version, both are connected to the same two peering IXP LANs and have a large number (several hundreds) of (almost identical) IPv4 and IPv6 sessions. Both routers have a connection to a different IP transit provider and are connected in the same way to the rest of our network. The routing engines' CPU load isn't flatline on 80+%, there are drops back to normal levels for minutes to hours, but these drops are not that often.

Things I've checked:

• no configuration changes have been made at the moment the increase started
• there's no increase in non-unicast traffic directed at the control plane
• there's no (substantial) change in the amount of traffic being forwarded (though even a increase shouldn't matter)
• show system processes summary indicates the rpd process is causing the high CPU load
• there are no rapidly flapping BGP peers causing a large amount of BGP changes

One possible explanation I can come up with is a peer (or more than one) on one of the IXP's both routers are connected to sending a large number of BGP updates. Currently I only have statistics on the number of BGP messages for my transit sessions (showing no abnormal activity) and with several hundreds of BGP sessions on the peering LANs it's not that easy to spot the problematic session(s) if I should create graphs for all sessions.

My questions are:

• are there any other things I should check to find the cause of this increase in CPU load on the routing engines?
• how can I easily find out which sessions are causing these problems (if my assumption is right)? Enabling BGP traceoptions generates huge amounts of data, but I'm not sure if it gives me any real insights.

Answer 1

There might be some helpful information for you at the Juniper Knowledge Center.

If RPD is consuming high CPU, then perform the following checks and verify the following parameters:

• Check the interfaces: Check if any interfaces are flapping on the router. This can be verified by looking at the output of the show log messages and show interfaces ge-x/y/z extensive commands. Troubleshoot why they are flapping; if possible you can consider enabling the hold-time for link up and link down.

• Check if there are syslog error messages related to interfaces or any FPC/PIC, by looking at the output of show log messages.

• Check the routes: Verify the total number of routes that are learned by the router by looking at the output of show route summary. Check if it has reached the maximum limit.

• Check the RPD tasks: Identify what is keeping the process busy. This can be checked by first enabling set task accounting on. Important: This itself might increase the load on CPU and its utilization; so do not forget to turn it off when you are done with the required output collection. Then run show task accounting and look for the thread with the high CPU time:

  user@router> show task accounting
  Task                       Started    User Time  System Time  Longest Run
  Scheduler                   146051        1.085        0.090        0.000
  Memory                           1        0.000            0        0.000  <omit>
  BGP.128.0.0.4+179              268       13.975        0.087        0.328
  BGP.0.0.0.0+179      18375163 1w5d 23:16:57.823    48:52.877        0.142
  BGP RT Background              134        8.826        0.023        0.099
  

Find out why a thread, which is related to a particular prefix or a protocol, is taking high CPU.

• You can also verify if routes are oscillating (or route churns) by looking at the output of the shell command: %rtsockmon –t

• Check RPD Memory. Some times High memory utilization might indirectly lead to high CPU.

Answer 2

I know this thread is old but for sake of completeness:

If the high cpu occurs randomly and you are not able to determine the process causing this we can create the script below.

With this script we are going to capture the process extensive when a process raises more than the normal or expected threshold, this should not disrupt any traffic but a MW is still recommended. However i see you have narrowed it to be RPD.

snmp {
    health-monitor {
        interval 30;
        rising-threshold 60;
        falling-threshold 50;
    }
}

event-options {
    policy MONITOR-CPU {
        events snmpd_health_mon_thresh_cross;
        attributes-match {
            snmpd_health_mon_thresh_cross.event-name matches "Health Monitor.+CPU.+rising";
        }
        then {
            execute-commands {
                commands {
                    "show system processes extensive";
                }
                output-filename cpu-processes;
                destination local-flash;
                output-format text;
            }
        }                               
    }
    destinations {
        local-flash {
            archive-sites {
                /var/tmp;
            }
        }
    }
}

DISPLAY SET OUTPUT>

set snmp health-monitor interval 30
set snmp health-monitor rising-threshold 60
set snmp health-monitor falling-threshold 50
set event-options policy MONITOR-CPU events snmpd_health_mon_thresh_cross
set event-options policy MONITOR-CPU attributes-match snmpd_health_mon_thresh_cross.event-name matches "Health Monitor.+CPU.+rising"
set event-options policy MONITOR-CPU then execute-commands commands "show system processes extensive"
set event-options policy MONITOR-CPU then execute-commands output-filename cpu-processes
set event-options policy MONITOR-CPU then execute-commands destination local-flash
set event-options policy MONITOR-CPU then execute-commands output-format text
set event-options destinations local-flash archive-sites /var/tmp

Also have you checked if any ddos messages have been reported? You could run the following commands:

show ddos-protection protocols statistics brief
show ddos-protection statistics
show ddos-protection version

Then depending what you see it can be narrowed down for example:

show ddos-protection protocols ttl statistics
show ddos-protection protocols ttl violations
show ddos-protection protocols ttl flow-detection detail  */*this cm needs prior config*/*

Juniper also has a collection list for this type of issues under KB22637

High CPU CLI Commands

set cli timestamp
show chassis routing-engine (multiple snapshots, atleast 5)
show system processes extensive (multiple snapshots atleast 5)
show system users
show system connections
show system statistics

Turn on task accounting and collect the task accounting detail output (three times with a gap of 30 seconds). Don't forget to turn it off after finished.

set task accounting on 
show task accounting detail
set task accounting off

show task memory detail
show task memeory summary
show task io
show task history
show task statistics
show task job
show task jobs
show krt queue
show krt state

Logs

Archive /var/log as specified in Step 1 above Traceoptions

user@router# show routing-options 
traceoptions { 
file routing-trace size 10m files 20 world-readable; 
flag task; 
flag state; 
flag timer; 
}

Also if you are running an old version which made be prone for bugs, you might want to check the life support of the code:

http://www.juniper.net/support/eol/junos.html

Another point to mention which could be a vector attack is not having protected your RE from unwanted exception traffic. Make sure you have a firewall filter under the loopback.

I have seen in the past scripts on the router causing high cpu not sure if rpd came into my view, but this is something you might not want to overlook.

If you see in the logs many hits with RPD_MPLS_PATH_BANDWIDTH_CHANGE you might be using a very aggressive adjust-interval

Check any drops on "show system queue: this is the kernel queue, some clue may appear.