I am troubleshooting a bizarre case of packet loss. We have a cabinet of servers with a top of cabinet switch (Brocade FESX648-PREM). That switch runs BGP sessions with our transit providers.
We have one server (referred to below as the "bad server") that's experiencing 50% packet loss. The server is running Windows Server 2012 R2 and it's been running for months without issue until this morning. At this point, I suspect something might be wrong with the switch itself, so I'm turning to this community for help with additional troubleshooting rather than ServerFault or SuperUser for server-related troubleshooting.
This is what I've checked so far to rule out the cause of the packet loss on the bad server:
- No other servers in the cabinet are experiencing packet loss.
- The gateway switch and bad server can ping each other without issue.
- If I log into another server in the cabinet and attempt to ping the bad server, then I do get the packet loss.
- The routing table on the bad server is fine -- the default route points to the proper gateway, no other entries exist (except for local IPv4 assignments).
- Firewalls have been disabled.
- No VPN setup is in effect (i.e., routing table on the bad server just has the default route).
- CPU load and network traffic are both very low.
- Server has been power cycled.
- Speed and duplex settings are set to auto-neg and are the same on both the switch and server.
- Forced 100mbit full on both ends, still had the packet loss.
- There are no port errors (no drops, collisions, FCS etc) recorded on the switch.
- CPU utilization on the switch is low (http://pastebin.com/q24QSqEz).
Anyone have any ideas where I should look next? The results of #2, #3, and #11 in particular are really throwing me for a loop...