I've been trying to diagnose internet connection issues at a remote small business for which I am the system administrator. If this is not the proper Stack Exchange to post to, I'd really appreciate it if you could direct me to the correct forum. I come in peace!
Every few minutes, the internet drops out and no computer in the office can get a proper connection. The topography is as follows. All devices are connected by gigabit ethernet.
Time Warner Business Modem
└── Apple AirPort Extreme AC
├── Netgear Unmanaged Gigabit Switch 1
│ ├── Apple iMac 1
│ │ └── Canon Printer
│ ├── Apple iMac 2
│ ├── Apple iMac 3
│ └── HP Ubuntu Laptop
├── Netgear Unmanaged Gigabit Switch 2
│ ├── HP Ubuntu Desktop
│ ├── Brother Printer 1
│ └── Brother Printer 2
└── Apple iMac 4
This problem began several weeks ago, and since then we have swapped the AirPort router for three other models, but the problem persisted.
To diagnose the issue, I've been running mtr (my traceroute) on several of the iMacs, pinging Google's Public DNS servers. I am also running mtr against the local network, having Apple iMac 1 ping HP Ubuntu Desktop, to try and rule out a local network issue with the wiring or Netgear switches. The local packets are never dropped, suggesting a fully stable connection. I'm including the IPv4 mtr session below. 10.0.0.1 is the AirPort's LAN IP, and the next IP address is Time Warner's server.
Packets Pings Host Loss% Drop Snt Last Avg Best Wrst StDev 1. 10.0.0.1 0.0% 0 10722 0.4 0.3 0.2 10.6 0.8 2. ##.###.##.## 7.9% 845 10722 22.1 21.7 7.7 292.0 14.2 3. ##.###.##.## 7.9% 845 10722 13.8 20.2 8.4 459.2 13.2 4. ##.###.##.## 7.9% 847 10722 22.3 22.0 8.9 374.5 13.2 5. ##.###.##.## 7.9% 852 10722 29.1 24.7 10.4 290.1 13.4 6. ##.###.##.## 7.9% 848 10722 23.3 25.2 8.3 643.9 15.3 7. 66.109.6.163 7.9% 849 10722 15.1 23.3 9.0 554.9 17.2 8. 66.110.96.53 7.9% 848 10722 26.3 21.3 7.9 467.2 14.4 9. 72.14.195.232 7.9% 846 10721 28.0 21.9 8.9 402.2 14.7 10. ??? 11. 108.170.238.201 7.9% 842 10721 19.1 21.8 8.8 498.9 14.5 12. 8.8.8.8 7.9% 844 10721 24.0 21.4 8.7 414.5 13.5
Every once in a while, the mtr sessions pinging Google DNS (on both IPv4 and IPv6) start dropping all packets for about a minute, after which everything starts working again. These periods of high packet loss coincide with the reported internet dropouts. This happens anywhere from 0-30 times per day. The packet loss begins with the first server beyond the router, in other words, the server that gives an IP address to the router from Time Warner Cable. Packets to the router are never dropped.
The Time Warner Modem is confirmed to be in bridge mode, and the AirPort successfully gets a WAN IP address, without any complaints (light is green, no double NAT, gets DNS servers, etc.). We pass all tests on test-ipv6.com.
Yesterday, we replaced the ethernet cable between the modem and the router because it appeared to be beaten up and kinked. The new cable is brand new, 10 gigabit with shielding, but the problem persists.
Last week, the mtr sessions reported only minimal packet loss, without any large periods of packet loss. The connection seemed stable. There doesn't seem to be any rhyme or reason to when it loses connectivity.
We have spoken with Time Warner Cable at length and they contend that they see no signal loss to our modem from their end, going back several months. Before this issue began several weeks ago, we have never had any issue with any of our equipment. All of the computers, routers, modems, and switches have been stable.
Now here's where it gets interesting — a month or so ago, the landlord installed a fire alarm backup power system in the same utility closet that houses the modem. It appears to power the strobe lights that would flash during a fire. The office manager reports that every few minutes, there is a distinct electrical clicking sound emanating from the power supply. I have not been on premises to witness this in person, but the office manager seems to believe that the clicking coincides with the internet dropout.
This is in part why I bought a shielded replacement cable, but it is my understanding that the two termination points have to be grounded for the shielding to work effectively, which they are not.
I am unsure of how to continue diagnosing this issue when there is no clear pattern. I am wondering if you know of any additional tests I should be running to rule out a local network issue (i.e. cabling, computers, router). I am comfortable with UNIX administration and can install and run whatever may be helpful (keep in mind the office is running macOS 10.13 and Ubuntu 14.04).
My second question is how likely is it that some sort of unknown electrical field is being discharged from the power supply, and if it is, is it more likely to be degrading the ethernet cable between the modem and router, or the modem itself? Is it worth putting together a makeshift Faraday cage to protect the modem? Should I be grounding the shielded cable on both ends? Or is this power supply a red herring? The router, I should note, is not in the same room, so is unlikely to be exposed to the same interference.
Unfortunately I am not able to travel to the office in the next few weeks, so I would prefer any advice on remote diagnosis.
Any ideas would be much appreciated. If anything is unclear, or you require more information, I'm happy to reply.
