Cisco SG500 52P CPU frequent CPU spikes

Question

We have two Cisco SG500 52PP model connected separately to our main NOC switch over fiber. Both switches are pretty much in standard configuration, VLAN1 is used, SNMP is enabled for remote monitoring. Very few users are connected on these switches and there is no load on the switches. We are observing CPU spikes after every few minutes and it touches 100% & ping delay variation occurs.

sw-mkt-12-206#sho cpu utilization
CPU utilization service is on.
CPU utilization
---------------
five seconds: 100%; one minute: 100%; five minutes: 59%

after a minute everything goes back to normal.

Following is Show Run config & related outputs ...

sw-mkt-12-206#sh running-config
config-file-header
sw-mkt-12-206
v1.4.10.6 / R800_NIK_1_4_214_020
CLI v1.0
set system mode switch queues-mode 4
file SSD indicator encrypted
@
ssd-control-start
ssd config
ssd file passphrase control unrestricted
no ssd file integrity control
ssd-control-end cb0a3fdb1f3a1af4e4430033719968c0
!
cdp device-id format hostname
voice vlan oui-table add 0001e3 Siemens_AG_phone________
voice vlan oui-table add 00036b Cisco_phone_____________
voice vlan oui-table add 00096e Avaya___________________
voice vlan oui-table add 000fe2 H3C_Aolynk______________
voice vlan oui-table add 0060b9 Philips_and_NEC_AG_phone
voice vlan oui-table add 00d01e Pingtel_phone___________
voice vlan oui-table add 00e075 Polycom/Veritel_phone___
voice vlan oui-table add 00e0bb 3Com_phone______________
errdisable recovery interval 30
errdisable recovery cause loopback-detection
errdisable recovery cause dot1x-src-address
errdisable recovery cause acl-deny
errdisable recovery cause stp-bpdu-guard
errdisable recovery cause stp-loopback-guard
errdisable recovery cause udld
no boot host auto-config
no boot host auto-update
hostname sw-mkt-12-206
logging host 10.0.0.1
logging source-interface vlan 1
no passwords complexity enable
username XXXX password encrypted e006122cbXXXXXXXX23edffea1a861
ip ssh server
snmp-server server
snmp-server location "Marketing Switch 12.206 - Installed @ Feb 2017"
snmp-server contact "zzzz"
snmp-server community MYSNMP ro view Default
clock timezone " " +5
clock source sntp
clock source browser
sntp unicast client enable
sntp unicast client poll
sntp server 10.0.0.2 poll
sntp source-interface vlan 1
!
interface vlan 1
 ip address 101.11.12.206 255.0.0.0
!
interface gigabitethernet1/1/23
 description "AP - MKT2 HALL - .12.227"
!
interface gigabitethernet1/1/47
 description "AP - MKT CORR .12.223"
!
interface gigabitethernet1/1/49
 description ULINK_2_NOC
!

sw-mkt-12-206#sh spanning-tree
Spanning tree enabled mode RSTP
Default port cost method:  long
Loopback guard:   Disabled

  Root ID    Priority    1
             Address     c4:b9:cd:85:f5:80
             Cost        20000
             Port        gi1/1/49
             Hello Time  2 sec  Max Age 20 sec  Forward Delay 15 sec
  Bridge ID  Priority    61440
             Address     00:8e:73:c0:93:a2
             Hello Time  2 sec  Max Age 20 sec  Forward Delay 15 sec

  Number of topology changes 392 last change occurred 38:25:04 ago
  Times:  hold 1, topology change 35, notification 2
          hello 2, max age 20, forward delay 15

after 1-2 minutes ping gets OK and CPU goes to normal & after few minute same repeats. & its happening constantly either in peak time, or in late night when no user is online. so I guess some process starts and it increases CPU and after it ends, the CPU goes normal.

I tried to upgrade firmware to latest version, but no avail.

any advise on How to track it?

Answer 1

Although it might not be the root cause of the symptoms you observe, there is one thing that triggers curiosity:

Number of topology changes 392 last change occurred 38:25:04 ago

Why? Did the L2 topology of your network actually change at that time?

In a stable network, STP topology changes should only ever occur when switches are added/removed from the network or if inter switch links change state.

So, last topology change times should appear as days and weeks, not hours, unless this is a "liquid" network with switches being added/removed every day.

Does show spanning-tree detail (if that, or a similar command is supported) reveal where the last topology change notification came from? Please update the question with CLI output that shows where the TCN came from.

By back-tracing the TCNs to their source, chances are good that you'll find a non-portfast enabled edge port (e.g. a server) that's just gone down/up.

Any non-portfast port going into forwarding state ("FWD") triggers a TCN being flooded across the given spanning-tree, upon which the switches will clear parts of their CAM tables. In turn, there will be a surge of unknown unicast flooding in your network (respectively: in that VLAN) until MAC addresses a re-learned.

--> make sure that ALL edge ports (clients, printers, servers, routers, Wifi APs, any device you are certain that is not bridging actively) in your network are set to spanning-tree portfast [trunk] resp. spanning-tree port type edge [trunk] (in Cisco speak - syntax may vary), best complemented by bpduguard (or the given platform's equivalent).