The Reverse Proxy Reloaded and Kept the Old Route Anyway
A hardening change on one of our platforms moved an API container off a shared Docker network so that only the reverse proxy could reach it. Sensible, standard, and the sort of change that looks risk-free because the proxy configuration was updated in the same commit.
Within minutes, live logins were failing with 502s.
The isolation was correct. The proxy configuration was correct. What was wrong was an assumption: that reloading the proxy would drop a route we had deleted from its configuration. It did not — the running process kept serving a direct route to an address that was no longer reachable.
We reverted, logins recovered, and the isolation went back on the list to be done properly.
Why a reload is not a restart
Most reverse proxies reload by re-reading configuration and applying the delta to the running process. That is the whole point: no dropped connections, no visible restart. It is also where the surprise lives, because what "applying the delta" covers varies by proxy and by what changed:
- DNS and upstream resolution is frequently cached. If an upstream name resolved to an address on the old network, that resolution can outlive the reload — so the proxy keeps dialling an address it can no longer reach.
- Connection pools to upstreams may be retained. Existing pooled connections are not necessarily torn down because the configuration describing them changed.
- Routes added at runtime — through an admin API, a service-discovery integration, or a container label — are not in the configuration file at all. Re-reading the file cannot remove them.
- Generated or included configuration may not be regenerated by a reload. The proxy re-reads what it is pointed at; if that is a file produced by another process, you have reloaded stale output.
Ours was the first and third of those together: the network change removed the path, and the route the proxy was still serving had not come from the file we edited.
Diagnosing it without guessing
The decisive question is what the running proxy believes, not what its configuration says. Ask the process, not the file.
If the proxy exposes an admin or config endpoint, read the live configuration back:
curl -s localhost:2019/config/ | jq '.apps.http.servers' # Caddy
nginx -T # nginx: dump the full effective config
nginx -T is the one people forget. It prints the complete configuration as loaded, with every
include resolved — which is exactly the thing you cannot determine by reading your own file.
Then prove reachability from inside the proxy's network namespace, not from the host:
docker exec -it proxy sh -c 'getent hosts api; wget -S -O- http://api:3000/healthz'
If the name resolves to an address the container cannot reach, you have the answer. A 502 means the proxy accepted the request and failed to reach the upstream — so the failure is always between proxy and upstream, never between client and proxy. That narrows it immediately.
The fix, and the safer pattern
The reliable remedy is a full restart of the proxy rather than a reload, when the change touches topology rather than routing:
docker compose up -d --force-recreate proxy
A reload is for "this path now goes to that path". A restart is for "the set of networks, upstream addresses or containers has changed". Topology changes invalidate cached resolution and pooled connections, and only a fresh process is guaranteed to start from the new truth.
Better still, order the change so there is never a window:
- Add the proxy to the new network while it remains on the old one.
- Restart the proxy; confirm it reaches the upstream by the new path.
- Only then remove the upstream from the old network.
- Verify a real request end to end.
Each step is independently reversible, and no step depends on a reload doing something you have not proved it does.
Verify with a real request
The check that would have caught this in seconds is a request through the real entry point for the one path that matters:
curl -s -o /dev/null -w '%{http_code}\n' https://app.example.com/api/auth/session
Not a health endpoint — health endpoints frequently bypass the routing you just changed. Pick the path that real users depend on. For us that was the auth route, which was precisely the one still being served directly.
Run it before the change, after the restart, and after the isolation. Three curls, each taking a second, bracketing a change that otherwise fails in production under real traffic.
What we took from it
Network isolation is worth doing and worth doing carefully. We reverted it rather than leaving logins broken while we investigated — the right call, and the revert is recorded in the history with the reason, so nobody re-attempts it without reading why the first attempt failed.
"Reload" is a performance optimisation, not a correctness guarantee. It exists so routing changes do not drop connections. It is not a promise that the running process now matches your file.
Ask the process what it thinks. Across proxies, service meshes, DNS resolvers and connection pools, the general bug is the same: a cached belief about where something lives, surviving the change that moved it. The configuration file is your intention. The running process is the fact.
