Mitigating a IPv6 malformed packet multicast flood

Question

So we just had our first IPv6 multicast flood in the network this morning. We managed to stop the offending computer by blocking the mac address with: mac-address-table static x.x.x vlan x drop

Before we blocked the mac address we started a Wireshark capture so we could analyze the packets later on.

After looking at the packets it seems the packets flooding the network was corrupt IPv6 DHCP requests contacting the IPv6 multicast address 33:33:00:01:00:02.

The impact of the flood was kind of weird, the only thing that seem affected was normal IPv4 DHCP requests, regular clients couldn't get a IP address, but those who already had one before the flooding started experienced no problems... The CPU's on the switches also peaked to 95-100%, but did not seem to impact normal traffic switching/routing operations.

What we need help to determine is how only 30mbps of IPv6 multicast traffic can push the CPU on a 6509 SUP720 to 100% and make normal IPv4 DHCP to stop working and how to protect ourselves against this should it happen again.

Each access/client port has the following configuration:

 switchport access vlan x
 switchport mode access
 switchport nonegotiate
 switchport block multicast
 switchport block unicast
 switchport port-security maximum 2
 switchport port-security
 switchport port-security aging time 2
 switchport port-security violation restrict
 switchport port-security aging type inactivity
 storm-control broadcast level 5.00 4.00
 storm-control multicast level 5.00 4.00
 spanning-tree portfast
 spanning-tree bpduguard enable

Isn't storm-control multicast applied to IPv6 multicast?

Here is a extract from the Wireshark capture with the "evil" packets on Dropbox.

And a little collection of graphs to illustrate the impact:

We also investigated the offending computer and could not find the reason or being able to reproduce the issue...

Answer 1

Background

I am somewhere that blocks Dropbox downloads, so I can't see the captured traffic, but I'll go on the assumption these were 64-byte ethernet multicasts.

Let's do some math to see how much traffic you're permitting...

A 64-byte frame is 672 bits (including 8 bytes for Preamble/SFD, and 12 bytes for IFG)...

8*(8 + 12 + 64) = 672 bits

That means line rate Gigabit Ethernet 64-bytes frames is about 1.488Mpps...

(1000000000 / 672.0) = 1488095.24 pps

What does this mean to you? Well your current storm-control configuration throttles traffic at 5% of line-rate, so you're allowing 74.4kpps of traffic to hit your switch before storm-control kicks in. 74.4kpps * 64-byte frames is 38Mbps, which is right at what your graphs show:

Answer

So, the bottom line is that you're allowing too much traffic to hit the switch CPU, which is why the CPU utilization was high. 74.4kpps is really too much to allow any switch CPU to process.

Assuming these stations shouldn't be sending a lot of multicast or broadast, the simple answer is to throttle your traffic like this...

   ! Note that broadcast traffic has the ethernet I/G bit set
   ! which means it is also classified technically as a multicast
   ! for storm-control purposes.  Therefore set your broadcast limit
   ! a little lower than your multicast limit
   storm-control broadcast level 0.4 0.3
   storm-control multicast level 0.5 0.3

Now storm-control kicks in when a port sends 6kpps of broadcast (0.4% 1GE line rate) or 7.5kpps of combined multicast / broadcast (0.5% 1GE line rate).

FYI, it is probably worth looking into Control Plane Policing, which protects the switch CPU against several ports ganging up on it. Keep in mind that CPP can be complicated to get right, so it's a good idea to test well before you roll it out in your environment.