Yesterday around noon, the internet at my company started acting up. No matter, slowdowns happen and there’s roadwork going on outside: maybe they hit the fiber or something. So we waited.
Then our Samba servers started getting flaky. And the database too. Uh oh… That’s different.
We started investigating. Some machines were dropping ICMP packets like crazy, then recovered, then other machines started to become unpingable too. I fired up Wireshark and discovered an absolute flood of IGMP packets on all the trunks, mostly broadcast from Windows machine. It was so bad two Linux machines on the same switch couldn’t ping each other reliably if the switch was connected to the intranet.
So we suspected a DDOS attack initiated from within the intranet by an outside attacker. We cut off the internet, but the storm of packets kept on coming. Physically disconnecting machines from the intranet one by one didn’t do a thing either.
Eventually, we started disconnecting each trunk one by one from the main router until we disconnected one and all the activity lights immediately stopped on all the ports. We reconnected it and the crazy traffic resumed.
So we went to that trunk’s subrouter and did the same thing. When we found the cable that stopped all the traffic, we followed it and finally found one lonely $10 ethernet switch with… a cable with both ends plugged into the switch. We disconnected the cable and everything instantly returned to normal.
One measly cable brought the entire company to a standstill for hours! Because half of the software we have to use are cloud crap or need to call their particular motherships to activate their licenses, many people couldn’t work anymore for no good technical reason at all while we investigated the networking issue.
Anyway, I thought switches had protections against that sort of loopback connection, and routers prevented circular routes. But there’s theory and there’s reality. Crazy!
There is such a thing as a small 1Gbps hub that are designed to just handle a small network. They scare me as they are cheap on Amazon and could theoretically bring a network to its knees if a random user finds a port that isn’t authenticated.
For the passers-by, in very simple terms:
A switch maintains a list of the IPs and MAC addresses of devices attached to it (ARP [Address Resolution Protocol] table). When a packet comes into the switch for a specific destination IP, the switch looks up on the ARP table where that destination IP can be found, and only sends the packet out on the port the destination device (or next hop towards that device) is connected to.
A hub doesn’t do any of that. Every packet that comes into the hub gets sent out of every port on the hub, to every device connected to the hub. It’s on the connected devices’ to discard packets that aren’t addressed to them. On anything but a very small and relatively slow network, this would create an unnecessarily large amount of traffic, not to mention the security issue around sending packets to devices they’re not addressed to.