Entirely wireless network being taken down by ICMPv6 neighbor solicitations

Question

We are in our second week of school, now, and starting yesterday we started receiving complaints about slow wireless internet. After some investigating, we found out that our entire wireless network is being sapped with ICMPv6 neighbor solicitation packets.

I captured a wireshark packet captures for 1 minute on multiple occasions, and it is pretty clear that this traffic is the culprit. During times where wireless is running flawlessly, about 1-2% of the total traffic is ICMPv6. When the wireless is bogged down, ICMPv6 accounts for 40-65% of the network traffic.

I'm running out of ideas as to how I can fix this. It looks like an IPv6 denial of service from inside. I can't pinpoint it to a single machine, as there are multiple IPv6 source and destination addresses, along with multiple source and destination MACs.

I have posted both a good and a bad sample of the traffic at https://www.dropbox.com/sh/u202fhol0t3tqtg/AAApKkm9PtbZwQfZtwkBViKka?dl=0

We thought we had this fixed after about 11:00 AM this morning, when we found an access point that was causing the issues. We replaced the AP with a new one and everything was fine for three hours until now.

Our wireless system is Unifi, by the way. 98% of our laptops are using Realtek 8188CE wireless NICs, and the other 2% are using a Centrino 7000 series NIC, not sure on the exact model.

I forgot to mention two things at the initial time of posting.

1) I have two buildings, only one of them is experiencing the problem. The networking equipment is identical between the buildings, and the only thing that is different is concurrent users (only about 70 less than the failing building), and laptop hardware, but the wireless cards are still the same except the working building is 100% Realtek 8188CE.

2) It will work during certain periods of the school day, then it will shut down for entire periods, which we thought pointed to malicious user activity, but any time we thought we were able to pinpoint it, the flood would start again.

UPDATE AS OF 8/28/14 11:14 AM

It currently appears that a piece of software installed, LAN School, used to monitor the activity of students, may have been the issue. We updated the version of this on every machine at the problematic building, and icmpv6 traffic has since maxed out at .5% and is often at 0%. I'm going to give it another day's time to verify, but it appears to be fixed.

Answer 1

It appears your network isn't running IPv6 -- no RAs are seen in the capture -- which may be a big part of the problem. Otherwise, it looks like the standard effects of a bunch of windows machines joining the network. Couple that with the simple fact that 99.999% of APs (wifi in general) don't handle multicast worth a damn, and you have the perfect recipe for a meltdown.

[** Multicast is just broadcast to most gear. In the wifi world, broadcast is handled at the "basic rate" which could be as low as 1Mbps. I've tested several APs (linksys, cisco, ...) with an HDHR as the source: it'll unicast two full video stream (39m) QAMs without a single error, but switch to multicast and even a 3M Standard Definition video sub-stream will kill wireless. (disclaimer) I've not done that with a Unifi.]

Alternate Option
1) Disable 802.11b -- use G only, or G/N only.
2) Change the basic-rate selection(s) on the APs. This will significantly reduce the range of each AP. And could cause some clients problems. But, it will increase the bandwidth for broadcast traffic.

(Have you talked to UBNT about your issues with IPv6? They may have nothing for you, but pinging them doesn't cost anything.)

Answer 2

The most recent update on my question was, indeed, the solution. It seems we had a few bad installs of LAN School on some (maybe all) of the laptops, and as soon as we updated it to the newest released version the problem disappeared entirely. Any time we see ICMPv6 traffic, sure enough, there is another laptop without the most up to date version of the software. Immediately upon updating it to the newest version, the ICMPv6 packets disappear again and we end up back to 0 ICMPv6 packets.

@RickyBeam Another update. The issue went away for a while, then just came back again yesterday afternoon. After more testing, it is a UBNT issue. We are putting our old HP Procurve wireless back in tonight. It remedies the problem entirely. This is the second time we have had to remove the Unifi product from our network.

Answer 3

MooseBalm, can you confirm, you are blocking RAs and the problem has been solved? It would be even better if you could kill them closer to the source, on the AP, or disable it on the clients completely.