SSH was up, the website wasn't: a bonding interface failure

28 July 2026

Monitors started firing for one of our servers, every alert pointing at the same box. First instinct is always to check if it’s actually down, so I SSH’d in over the internal network. No problem. Logged in clean, load looked normal, nginx was running, the app responded fine when I curled it locally.

But every request coming in through Cloudflare to the public site was timing out. Not slow, timing out completely. That mismatch is what mattered. If the box were down or the app had crashed, SSH wouldn’t have worked either. Something was breaking specifically on the path between the outside world and this server, not inside it.

That pointed at the network layer, not the app layer. The common thread between “internal SSH works” and “outbound facing traffic doesn’t” is that the two use the interface differently. That meant the bonded NIC setup was the next place to check, not the webserver config.

The fix

Once I got there, this cleared it:

modprobe -r bonding
modprobe bonding

That removed the bonding kernel module and reloaded it fresh, then I brought eth0 and the bond interface back up. Traffic through Cloudflare started resolving again almost immediately. Ops was looped in as soon as the fix was confirmed, since a network layer failure on a prod box needs to be on record no matter how fast it gets resolved.

What I think actually happened

I’ll say this plainly: I fixed the symptom fast, but I don’t have hard proof of the root cause yet, and I’d rather be honest about that than pretend otherwise.

My working theory is that the bonding driver got stuck in a bad internal link state, the kind of thing where the MII monitor still reports a slave as up even though it isn’t actually forwarding traffic correctly anymore. A soft interface bounce wouldn’t have cleared that, since the driver’s own state machine was the problem, not just an up or down flag. A full module reload forces the driver to rebuild that state from scratch, which lines up with why the reload worked and a simple ifdown/ifup likely wouldn’t have.

That also fits the SSH versus Cloudflare split. My SSH session didn’t need a fresh forwarding decision through the bond, since the switch already had the MAC cached for that LAN path. Every new outbound flow toward Cloudflare, on the other hand, needed the bond to correctly pick a working slave, and that’s exactly the part that seems to have been stuck.

What I still want to check before calling this fully closed: pull journalctl -k from the incident window and look for NETDEV WATCHDOG transmit timeouts or repeated link flap events on eth0 right before it broke. If those show up, that points to a driver or firmware level hang rather than a clean physical fault, which changes whether this is something to just monitor or something to raise with whoever owns the hardware.

Commands worth having ready

If you’re chasing something similar, these are the ones I’d reach for first, in the order I’d actually run them.

Check current bond and slave state:

cat /proc/net/bonding/bond0

Shows which slave is active, MII status per slave, and failure counts. This is the first thing to check, before touching anything.

Check interface link and carrier state:

ip link show

Confirms whether the kernel thinks the interface is up and has carrier, separate from what the bonding driver reports.

Check the physical link at the driver level:

ethtool eth0

Shows actual link detection, speed, and duplex. Useful for telling a driver level issue apart from a real cable or switch port problem.

Check for errors or drops on the interface:

ip -s link show eth0

RX and TX error or drop counters climbing here point toward a hardware or driver issue rather than a routing or firewall problem.

Check kernel logs around the incident window:

journalctl -k --since "1 hour ago" | grep -i bond

Look for NETDEV WATCHDOG, link flap messages, or driver reset events. This is the piece I still need to go back and confirm for this specific incident.

Safe recovery sequence, if a reload is genuinely needed:

ip link set bond0 down
modprobe -r bonding
modprobe bonding
ip link set eth0 up
ip link set bond0 up

Bring the bond down cleanly first rather than yanking the module while it’s still active. Confirm with cat /proc/net/bonding/bond0 afterward that both slaves rejoin correctly before considering it resolved.

Fixed fast, documented honestly, root cause still half open. That’s the real state of it, and probably the most useful part of this post if you ever land here searching for the same symptom.