7

So we just had our first IPv6 multicast flood in the network this morning. We managed to stop the offending computer by blocking the mac address with: mac-address-table static x.x.x vlan x drop

Before we blocked the mac address we started a Wireshark capture so we could analyze the packets later on.

After looking at the packets it seems the packets flooding the network was corrupt IPv6 DHCP requests contacting the IPv6 multicast address 33:33:00:01:00:02.

The impact of the flood was kind of weird, the only thing that seem affected was normal IPv4 DHCP requests, regular clients couldn't get a IP address, but those who already had one before the flooding started experienced no problems... The CPU's on the switches also peaked to 95-100%, but did not seem to impact normal traffic switching/routing operations.

What we need help to determine is how only 30mbps of IPv6 multicast traffic can push the CPU on a 6509 SUP720 to 100% and make normal IPv4 DHCP to stop working and how to protect ourselves against this should it happen again.

Each access/client port has the following configuration:

 switchport access vlan x
 switchport mode access
 switchport nonegotiate
 switchport block multicast
 switchport block unicast
 switchport port-security maximum 2
 switchport port-security
 switchport port-security aging time 2
 switchport port-security violation restrict
 switchport port-security aging type inactivity
 storm-control broadcast level 5.00 4.00
 storm-control multicast level 5.00 4.00
 spanning-tree portfast
 spanning-tree bpduguard enable

Isn't storm-control multicast applied to IPv6 multicast?

Here is a extract from the Wireshark capture with the "evil" packets on Dropbox.

And a little collection of graphs to illustrate the impact:

CPU Spike Graph

We also investigated the offending computer and could not find the reason or being able to reproduce the issue...

7

Background

I am somewhere that blocks Dropbox downloads, so I can't see the captured traffic, but I'll go on the assumption these were 64-byte ethernet multicasts.

Let's do some math to see how much traffic you're permitting...

A 64-byte frame is 672 bits (including 8 bytes for Preamble/SFD, and 12 bytes for IFG)...

8*(8 + 12 + 64) = 672 bits

That means line rate Gigabit Ethernet 64-bytes frames is about 1.488Mpps...

(1000000000 / 672.0) = 1488095.24 pps

What does this mean to you? Well your current storm-control configuration throttles traffic at 5% of line-rate, so you're allowing 74.4kpps of traffic to hit your switch before storm-control kicks in. 74.4kpps * 64-byte frames is 38Mbps, which is right at what your graphs show:

Answer

So, the bottom line is that you're allowing too much traffic to hit the switch CPU, which is why the CPU utilization was high. 74.4kpps is really too much to allow any switch CPU to process.

Assuming these stations shouldn't be sending a lot of multicast or broadast, the simple answer is to throttle your traffic like this...

   ! Note that broadcast traffic has the ethernet I/G bit set
   ! which means it is also classified technically as a multicast
   ! for storm-control purposes.  Therefore set your broadcast limit
   ! a little lower than your multicast limit
   storm-control broadcast level 0.4 0.3
   storm-control multicast level 0.5 0.3

Now storm-control kicks in when a port sends 6kpps of broadcast (0.4% 1GE line rate) or 7.5kpps of combined multicast / broadcast (0.5% 1GE line rate).

FYI, it is probably worth looking into Control Plane Policing, which protects the switch CPU against several ports ganging up on it. Keep in mind that CPP can be complicated to get right, so it's a good idea to test well before you roll it out in your environment.

| improve this answer | |
  • Great answer Mike, I wish I could up-vote this more than once! I've run into this situation several times, where people underestimate just how little traffic it takes to hobble the CPU in a router/switch. People never seem to remember packets per second, they focus on bits per second. (I'm guilty of it too.) – Brett Lykins Nov 21 '13 at 15:55
  • 1
    Thanks Brett, I think if Cisco did this right we could apply storm control to traffic in either pps or bps on any platform – Mike Pennington Nov 21 '13 at 16:26
  • Thanks for all the answers so far, so by going by this answer; storm-control multicast should also apply to IPv6 traffic? And further, It seems like it's better to use storm-control with pps rather than mbps, as that is more predictable... – mastrboy Nov 25 '13 at 18:44
  • Yes, storm-control applies to IPv6 traffic, because storm-control is just looking at the ethernet header. IPv4 / IPv6 mcast both set the I/G bit in the ethernet header. While it's better to use pps as opposed to Mbps in the storm-control command, Cisco only gives you the option of % line rate on some models. Just remember to calculate how many pps of traffic you're allowing the CPU to process when you set storm-control limits – Mike Pennington Nov 25 '13 at 18:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.