Teng Li · ML Engineer

Engineering field notes

Every Layer Lied to Me

Debugging a remote OpenWrt install.

A laptop connects to an OpenWrt board through an uncertain network path. Schematic illustration.
On this page

I nearly blamed a perfectly good SD card for a remote OpenWrt install.

By then I had a blinking heartbeat LED, a live Ethernet link, an admin page that appeared intermittently, and an SSH window that stayed open through a power cut. I had collected plenty of observations. I kept asking them to prove more than they could.

The job was small: flash OpenWrt onto a NanoPi R2S, assign a static address, and connect it to a VPN overlay. I was working through a remote session on a macOS laptop with a direct Ethernet connection to the board.

The board was fine. The interesting failures were in the path I used to reach it—and in the conclusions I drew from that path.

The setup

A macOS laptop has two separate links: Wi-Fi to an existing gateway and direct USB Ethernet to the NanoPi LAN port. Both links use the example subnet 192.168.1.0/24, and both remote peers have address 192.168.1.1.
01 Management wiring after moving the cable to LAN. Separate links, overlapping example addresses. Full-size ↗

01A live link on the wrong port

After a reflash, the cable ended up in the WAN port.

In the configuration I was using, management access was on LAN. WAN expected a DHCP lease and restricted incoming traffic. On the direct cable, there was no DHCP server to provide that lease.

The heartbeat LED blinked. Ethernet had carrier. The management address stayed unreachable.

That combination made the board look broken because I had quietly treated “the link is up” as “the management interface is connected.” The LED had never promised that.

The mistake was ordinary router muscle memory: an uplink goes into WAN. For this connection, I needed the port assigned to LAN. Checking which physical port carried the cable was more useful than another round of address guessing.

02Two networks, one destination

Moving the cable exposed the next problem. The laptop had Wi-Fi on the existing LAN and USB Ethernet connected to the board. Both networks used 192.168.1.0/24.

Both also contained a device at 192.168.1.1: the existing gateway on Wi-Fi and the freshly flashed board on Ethernet.

I could get an answer from .1 and still have learned nothing about the board. Requests succeeded, then failed, while I kept describing the target as intermittent. My test machine had two possible destinations for the address I was typing.

A host route toward the wired connection helped make the path explicit. Even that had a prerequisite: the board had to be powered and the link available. An earlier route-add attempt returned “Network is unreachable.”

I also stopped treating an interface option as proof of egress. OpenSSH documents ssh -B as binding to an interface's address. curl's --interface accepts interface names or addresses. Those options deserve a platform-specific reading; a successful request still needs a route and an identified peer.

“Something answered” was the weakest possible success criterion for this setup.

03My fixes became part of the fault

While working on the host route, I found a permanent ARP entry mapping the board's address to the USB adapter's own MAC address.

During cleanup, I also ended up with a proxy entry associated with Wi-Fi. Attempts to delete what looked like the offending entry kept returning “cannot locate.” I was looking at the destination address while missing the entry's scope and type.

Apple's arp manual distinguishes interface-scoped and published entries. Multiple entries can share a destination across interfaces; deleting one does not establish that every relevant entry is gone.

Once I accounted for those distinctions, I could remove the entries in the right order. Until then, each repair attempt added state I had to understand before I could interpret the next result.

The laptop had become part of the incident. I was still debugging as though it were a neutral instrument.

04A login page is a weak fingerprint

Both the gateway and the board served admin pages. I misidentified their responses by looking at page source.

My first shortcut was head -c 200. The board's landing page was a redirect stub, and the useful marker appeared after the part I had kept. I had thrown away the evidence before inspecting it.

Searching the full response for cgi-bin/luci was more useful. A diagnostic version that keeps a failed fetch visible is:

curl -sS -m 5 http://192.168.1.1/ -o /tmp/openwrt-page.html &&
  grep -c 'cgi-bin/luci' /tmp/openwrt-page.html

That narrows the question to “does this response contain the LuCI marker?” A match helped distinguish the two pages in this setup. It does not authenticate a particular board. A zero count can mean a different page; a failed fetch needs its own diagnosis.

Ping was equally easy to overread. The board did not answer the probes I tried. One apparent success reached the gateway over Wi-Fi instead. An ICMP reply established reachability to a responder. I still had to establish which responder it was, and whether the service I needed worked.

05Configuration, service, session

Getting to the right host left several more ways to mistake recorded state for current behavior.

SSH was enabled in configuration but wasn't accepting connections. Restarting the service through LuCI restored access. A refusal and a timeout were different observations: refusal indicated an explicit rejection somewhere on the path; timeout meant I had no timely answer. Neither, by itself, identified the machine or proved exactly where the failure occurred.

A reflash changed the SSH host key. In this setup, where two machines could occupy the same destination address, dismissing that warning would have been especially careless. OpenSSH's host-key check exists to detect changed identity. Replacing a stale entry belongs after verifying the intended device and its new key through a trusted path.

An open SSH window survived a power cut. An idle connection can remain on screen until traffic, keepalives, or a timeout exposes the loss. I needed a fresh login and the board's uptime to establish that it had rebooted. An unchanged terminal window was a poor substitute.

Network settings rolled back after a GUI apply. LuCI has a confirmation-and-rollback mechanism: an unconfirmed apply can revert. I had treated the submitted settings as the active settings. Applying the intended configuration through the CLI, reading it back, and then checking it after a reboot gave me a clearer record of what had persisted.

Each failure had the same shape. I was using one state—configured, displayed, submitted—as evidence for another—running, connected, persisted.

06Two different packet paths

The final problem appeared after the board had joined the overlay. The tunnel was active and its counters moved, but application traffic sent through it still failed.

The board itself had downloaded packages and reached the overlay. That success covered traffic originating on the board. Forwarded traffic had another path through the firewall.

Requests using a numeric IP failed too. That gave me a reason to look beyond DNS, although a numeric-IP failure alone could not locate the fault.

Two simplified packet paths on the board. A board-originated request passes local output policy. An overlay packet crosses forwarding policy, which blocked the unassigned tunnel before a zone and forwarding rule were configured.
02 Local output and forwarding follow different paths. Zone, masquerading, and MSS settings changed together. Full-size ↗

The tunnel device was outside the firewall zones I had configured, and the forwarding policy blocked the path. OpenWrt's firewall configuration separates zone membership, forwarding, masquerading, and MTU adjustment. A working tunnel does not establish permission to forward its packets.

The working configuration gave the tunnel a dedicated zone, allowed the intended forwarding to LAN, and enabled masquerading and MSS clamping for the path. Restarting the firewall restored traffic.

I changed those settings together, so I cannot claim this isolated an MTU problem or proved each setting necessary. The concrete configuration gap was the missing zone and forwarding path. The restored application request was the useful final check.

What the measurements actually proved

At the lowest point, I suspected the SD card. The faults I found were a cable in the wrong port, overlapping networks on my test machine, misleading ARP state, and a firewall path I had not configured.

I had run into a related mistake when my agent eval marked correct behavior as failure: the measurement answered a narrower question than the conclusion I attached to it.

Six signals I kept asking to prove too much

Link light
EstablishedPhysical carrier
Still to checkCorrect management port
HTTP at .1
EstablishedA responder
Still to checkWhich peer answered
LuCI marker
EstablishedRecognizable page content
Still to checkAuthenticated device identity
Open SSH window
EstablishedA client session remained
Still to checkCurrent peer liveness
Submitted settings
EstablishedAn intended configuration
Still to checkActive and persistent settings
Tunnel counters
EstablishedTraffic at the tunnel
Still to checkEnd-to-end forwarding

The runbook I kept from this install now asks for four things:

  1. Establish the path. Physical port, selected route, and neighbor entry come before a hardware diagnosis.
  2. Identify the responder. Record what an address actually reached.
  3. Name the claim. Service availability, saved configuration, reboot, and forwarded traffic each need their own check.
  4. Change one thing, then measure again. A repair that adds unexplained state makes the next result harder to interpret.

The board has since survived reboots and power cuts. A fresh connection works, the intended configuration persists, and application traffic passes through the configured path.

Those are smaller claims than “everything looks healthy.” I can say how I checked each one.

← All writing