> ./SILENT_INGEST_FAILURE
My hardened lab environment refused every agent request for 35 hours. Nothing alerted, and the one health signal anyone would have checked stayed green the whole time. The networking bug was the smaller half of this. The monitoring gap was the real finding.
Symptom
Every request to the lab server came back 403 from nginx. More than 23,000 denials accumulated before anyone noticed, and the way it surfaced was not a monitor firing. I tried to log in to the console and could not.
Why it stayed invisible for 35 hours
This is the part I care about. A misconfiguration that announces itself is an inconvenience. One that hides behind a healthy dashboard is a detection problem, and three separate gaps had to line up for it to hide this well.
No success signal to miss
nginx emits no access log in this profile. There was no record of healthy requests, so their absence could not be noticed by anyone or anything.
Nothing watched the denial rate
Denials were being written. No threshold, no alert, no dashboard looked at them. The evidence existed the entire time and nobody was pointed at it.
The one green signal was the misleading one
A metrics WebSocket opened two days before the outage stayed open throughout. Because it never reconnected, it never re-entered the broken path. Every “is the agent alive” check kept returning healthy while durable ingest was fully blocked.
Root cause
The lab authorizes by mesh VPN address. Each nginx location carries an explicit allow list and a closing deny all. A peer's packet reaches the web container by being DNAT'd from the host's mesh address to the container inside a bridge network, and is then forwarded into that bridge.
Because it arrives as forwarded traffic on the mesh interface, Tailscale marks it 0x40000, and its ts-postrouting chain masquerades on that mark. The source address is rewritten to the bridge gateway. nginx then saw one identical address for every client on the network and matched no allow rule.
The distinction that matters: the ACL was never bypassed. The information it authorizes on was erased before it got there. The failure was closed rather than open — which is the correct direction to fail, and is also exactly why nothing looked alarming.
The trigger was a service restart during routine unattended update activity. First denial landed 2 minutes 46 seconds later. I can prove the masquerade was the cause by removing it; I never established precisely what the restart changed, and I would rather say that than invent a tidier ending.
Fix and verification
Disabling source NAT for subnet routes removes the offending masquerade rule. I verified by effect rather than by assumption: denials stopped, three fresh connections opened with keepalive disabled — so they could not ride an existing session — returned 200, proving new connections carried a real peer address again, and a 12-minute soak recorded zero denials with ingest present in 12 of 12 minutes.
This setting is a deliberate deviation from my certified baseline, so it is recorded as one. It survives restarts, but bringing the mesh up with different flags would silently revert it.
What changed as a result
The runtime start gate gained a check with two halves. One asserts that no masquerade rule can rewrite an inbound peer address and that the expected DNAT publication exists. The other asserts that no recent denial is sourced from the bridge gateway — the exact fingerprint this outage left. A test profile guards both halves and the call site, so the check cannot quietly stop being called.
I wrote the gate against the fingerprint rather than the trigger on purpose. I still do not know what the restart changed, so a check that watched for that specific restart would protect me against one cause of a class of failures.
The rule I took from it
> Liveness on a long-lived connection is not evidence that the request path works.
An established stream is not re-evaluated against an ACL on every message. A per-request path is. They do not share a failure domain, so one cannot stand in as the health check for the other — and the stream is the one that keeps looking fine.
The second, less comfortable lesson: I had no success signal anywhere in this system. Turning off access logging is a reasonable hardening choice on its own, but combined with no alerting it left an environment where healthy and completely broken produced identical observable output. I would take the log volume over that trade again.