Anton, chapter 11: The loop closes on itself

2026-06-26 → 2026-07-16

The model flies now, and I still ship every fix by hand. That is the itch I keep returning to. A change lands in the repo, tests go green, and then it sits there until I open a terminal, pull on the home server, rebuild, and watch the health check. The intelligence got fast; the delivery stayed manual. So the question I want to answer is whether the loop can close all the way (issue, review, merge, deploy, running on the family's phones) with nobody in the middle, and without ever shipping something broken to the five people who actually depend on it. The safety half is the whole point. A pipeline that deploys fast and deploys wrong is worse than my hands.

The spine goes in first: a CI gate that will not let anything merge until typecheck, build, and the full test suite are green, and a deploy job that reaches the home server over the private tailnet and runs the health-gated update script I already trust (pull, build, probe health, roll the image back on its own if the probe fails). Keeping the deploy path inside the tailnet means production never opens an SSH door to the public internet; the runner joins the network the same way a laptop would. That part is the boring kind of plumbing that either works or embarrasses you, and it works. (Almost all of it: the deploy notifications turn out to be wired to a channel whose credentials were never set, reporting nothing to no one until I reroute them. Even the messenger needed a health check.)

The smoke gate is where I get educated. The idea is small: after a deploy, ask the live system a handful of questions and only keep the new version if it answers them. The execution fights me for three weeks. First the gate calls an endpoint that isn't there, reads the 404 as a failure, and rolls back every single deploy while paging me CRITICAL. I fix the endpoint and it rolls back again, this time because one of my own test cases asserts that a conversation-search query should route to the knowledge domain when it correctly routes to memory: the gate was wrong, the system was right, and the gate won anyway. I fix the assertion and hit the third wall, the quiet one: six model calls take longer than the timeout, so the gate is silently skipped on every deploy. It has been green the whole time because it never ran. That is the worst failure mode there is. A gate that false-fails gets angry attention and gets fixed; a gate that silently skips reports confidence it never earned, and you find out the day it was supposed to save you.

So I make it fit its own budget. The cases run concurrently instead of one after another, I trim the set until it returns a real verdict inside the window, and then I split the checks into two tiers so the trimmed-back cases come home without power to block: a hard tier that can roll a deploy back, and a soft tier that reports but doesn't. The gate earns its keep for the first time a few days later, when it refuses a commit that genuinely broke two family-facing paths and rolls it back on its own. I didn't catch that one. The thing I built to not trust me caught it, and that is exactly the day the work paid for itself.

The last piece is the fleet that does the building. The sessions that pick up issues and review pull requests are long-lived, and after a full day their context saturates and they start getting slow and strange. So there's a watchdog that notices a stuck session and kills and respawns it with a fresh head, and a supervisor that sleeps the whole fleet when the issue queue is empty and wakes it the moment something new is marked ready. It is a small piece of stagecraft, but it's the difference between an automation that runs for an afternoon and one that runs for weeks.

Where it lands is quieter than I expected. I file an issue, describe the fix I want, and walk away. Some hours later the change is live on the family's phones, or it isn't and nothing shipped, and either way I didn't touch the server. The loop closes on itself. What I notice isn't the speed. It's that the piece I trust most in the whole pipeline is the one whose only job is to stop it.