5

We had a broadcast storm that went undiagnosed for several hours. After the problem switch was unplugged and traffic returned to normal, we had a handful of machines and switches that were broken. On one switch, the sole uplink interface had to be moved to another int. On another, one of the two etherchannel physical members is down. One machine has a SSD and it didn't boot up after it was shut down. BIOS says it's a 32 KB disk. At least one other server's fans would ramp up to high RPMs every couple of minutes.

I read that storm traffic not intended for a machine is dropped by the NIC, but broadcast traffic is sent up the network stack and can cause high CPU utilization. I imagine if the OS was writing log files because of the increased network activity it could eventually fill up disk space and/or burn up an SSD because of the increased read/writes.

Improve this question
2
  • IMO, looks like Energy or EMI problem... broadcast strom is maybe the consequence, not the root cause. – Golgot Dec 27 '17 at 15:56
  • 1
    Did any answer help you? If so, you should accept the answer so that the question doesn't keep popping up forever, looking for an answer. Alternatively, you can post and accept your own answer. – Ron Maupin♦ Jan 5 at 17:20
5

I don't think this is possible. A broadcast storm from a bridge loop will effectively bring down much/all of your network (depending on bandwidth) but doing some permanent damage to the hardware is hardly imaginable unless cooling is barely working. Exhausting TBW on an SSD is also hardly likely - maybe with packet capture and a ring buffer...

Could you find out what exactly happened? Is it possible the cause wasn't a bridge loop but a hacking attack? A hacker covering up his tracks?

Improve this answer
4
  • As soon as I unplugged one of the switches involved, the storm stopped. If it were a hacker, they would've had to be physically in the building. The behavior didn't resurface when I plugged the switch back in. The people serviced by the switch are in and out of the building all day, so I suspect one of them plugged the wrong cable in somewhere and created the loop. Even if their laptop were on the wireless network, a storm would've prevented traffic on that NIC too. – user208145 Dec 26 '17 at 23:50
  • When you say you "unplugged the switch", are you referring to power, or a subset of the network cables plugged into it? If the former, that should not have resolved the loop, and the storm should have resumed. If the latter, did you restore all of the cables you unplugged? If not, do you know which of the remaining cables caused the loop? – Slartibartfast Mar 27 '18 at 4:34
  • I pulled power and left the switch off for the remainder of the business day. At the core switch, I shut down VLANs to drop the traffic. By the time we surmised what happened, there were no physical loops that we could find in the department serviced by the switch I powered off. Some people were moving offices that day, so I have a general idea who did it and which ints they were. – user208145 Mar 28 '18 at 19:33
  • Well, possibly there was a remote loop - someone connected a switch to an edge port and caused a loop there. There are many possibilities but unless you did find the cause you'll never know... – Zac67 Mar 28 '18 at 19:55
2

Zac67 answer is spot on, but I would emphasis on the temperature issue.

The fact is the main impact of the storm on the hardware is the load, which increase the temperature of the components. If cooling is inadequate it could, in extreme cases, damage the hardware.

Most enterprise-grade devices have at least one internal temperature sensor, so I would check if those are properly monitored and what are the current temperature as well as the increase during the incident. An overheating should also be notified in the device logs.

Improve this answer
1
  • Temperature is good. I have a weekly report coming to me. During the storm I didn't see any errors about temperatures being exceeded. – user208145 Mar 28 '18 at 19:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.