How Taxiway deploys without dropping a request

“Zero-downtime deploys” is on the feature list of every platform on earth, including this one. Most of the time nobody checks. This is the bit that actually goes wrong, why it goes wrong, and the unglamorous fix, plus the test that proves it, because a claim like this without a test is just a nice sentence.

What a deploy actually is

You push. A webhook reaches the control plane, which pins the commit, builds a new desired state and hands it to the agent on your server. From there the agent does four things.

1. Build

The agent clones your repository at the pinned ref and builds an image with Docker. Not the control plane. Your server, using your CPU. Your source never lands on our disks.

A build is worth being paranoid about. It is the least trusted and least predictable thing that will ever run on the host: it is arbitrary code from a repository, invoked by whoever can push to it, on a machine that is also running your production database. So a build runs under explicit limits. A runaway npm install should fail a deploy, not take down everything else on the box.

2. Start, but don’t route

The new container starts alongside the old one. Both exist. Only one is serving traffic, because the new one is not in the load balancer’s pool until its readiness probe passes.

If it never passes, the deploy fails and the old container is still there, still serving, completely untouched. A failed deploy should be a non-event.

3. Overlap, the interesting part

Now the swap. And here is the thing nobody tells you about label-driven routing.

Taxiway uses Traefik at the edge, configured entirely through Docker labels. Both the old and new containers carry the same service labels, so for a moment they are two backends in one load balancer pool. That is the correct way to do a cutover: the pool is warm, requests are being served by both, and you drain the old one.

Except you can’t drain it. Traefik discovers backends by watching Docker, and labels give you no way to deregister a backend before you stop it. The sequence is forced:

1. You stop the old container.
2. Some milliseconds later, Traefik notices it is gone and removes it from the pool.

Between those two points, the pool contains a backend that is no longer listening. Any request that Traefik load balances onto it gets a connection refused, which Traefik turns into a 502 Bad Gateway and hands to your user. The window is small. It is not zero, and “small” is not the same as “never” when you are serving real traffic.

4. Reap

Once the new container is healthy and routed, the old one is stopped and removed on the reaper’s next pass, which comes round every thirty seconds and deletes any managed container that is not in the desired set.

That reaper is also why transient containers carry a role label. When a backup dump is running, it is a container the agent created that is deliberately not part of the desired state, and without a label saying “this is a system job, leave it alone” the reaper would cheerfully destroy it thirty seconds into a database dump.

Closing the window

So: you cannot pull a backend out of the pool before stopping it, and you cannot stop it without briefly leaving a dead one in. There is no ordering that avoids the gap.

You can, however, decide what happens when a request lands on it. Traefik has a retry middleware: if a backend refuses the connection, re-send the request to another one in the pool. Because the pool has just been made to contain the new, healthy container, the retry lands there.

That turns the failure mode from some users get a 502 into some users get a request that took a few extra milliseconds. That is the whole fix. It is one middleware on the entrypoint, and it is not optional: it is the only thing standing between a normal deploy and a handful of error pages.

Proving it

A claim like that is worth nothing without a test, so there is one, and it runs against real Docker rather than a mock. It:

1. Creates a network and starts two containers with identical pool labels, exactly what the agent produces mid-cutover. One is a working web server. One is sleep 600: it exists, so Traefik registers it, but it will never accept a connection. That is a dead backend, deterministically.
2. Starts two Traefik instances against that network: one with the retry middleware, one without.
3. Waits until both Traefiks report a pool size of two. This step matters more than it looks: if the test proceeded with only the live container registered, everything would pass and the test would prove nothing.
4. Fires twenty requests at each.

Without retry, some of those twenty come back 502. The test asserts that they do: if no request reaches the dead backend, the test fails as inconclusive rather than passing, because a test that can silently stop testing the thing is worse than no test.

With retry, all twenty come back 200.

Rollback falls out for free

Deployments are append-only. Every deploy is a row, its spec is kept, and the resource points at whichever one is active. That spec is the applied state; there is no separate “applied” column that could drift from it.

Rolling back therefore isn’t a special code path. It sets the target back to an old deployment’s spec and rolls forward through exactly the same cutover described above, retry middleware and all. The same machinery, pointed at an older image.

This is the part that host-provisioning panels struggle to match. If your deploy consists of git pull plus a service restart on a server that has been apt-upgraded forty times since the commit you are rolling back to, then “roll back” means “run old code against whatever state the host is in today”. With an image, the thing you roll back to is the thing that was running.

What is still missing

Being straight about the gaps: a project runs on one server today. There is no scheduling of one project across several machines, so this is zero-downtime deployment, not high availability. If the host dies, the site is down until it comes back. Multi-server clustering is a thing I want to build and have not built.

---

The retry middleware is four lines of Traefik configuration. Finding out it was necessary took a production 502 and an afternoon with docker events. That is most of what this job is.