[{"data":1,"prerenderedAt":4321},["ShallowReactive",2],{"writing-\u002Fideas\u002F2026-07-14-anton-12-total-recall-lie":3,"all-writing":58,"related-work-\u002Fideas\u002F2026-07-14-anton-12-total-recall-lie":4286},{"id":4,"title":5,"body":6,"canonical_url":38,"chapter":39,"date":40,"description":15,"extension":41,"featured":42,"image":38,"kind":43,"meta":44,"navigation":45,"path":46,"seo":47,"series":48,"stem":49,"summary":50,"tags":51,"work_slug":52,"__hash__":57},"writing\u002Fideas\u002F2026-07-14-anton-12-total-recall-lie.md","Anton, chapter 12: Total recall was a lie",{"type":7,"value":8,"toc":34},"minimark",[9,16,19,22,25,28,31],[10,11,12],"p",{},[13,14,15],"em",{},"2026-07-14 → 2026-07-22",[10,17,18],{},"Two claims sit at the center of what Anton is supposed to be: it remembers everything I tell it, and it knows how to use all of its tools. One evening I take a single screenshot of a normal exchange on my phone, and both claims fall over inside the same hour. That is the whole shape of what follows. Not a hard bug hunt, a humbling. The features had shipped, the capabilities were loaded, and nobody had checked whether either one actually did the thing it promised on the live box.",[10,20,21],{},"The first crack is a request I've made a hundred times in my head and apparently never once on the phone: add something to the family calendar. Anton tells me it can't add events directly and offers me a reminder instead, politely, as if that were a real answer. But the create-event capability exists; I built it. The trace tells the story in four lines. The family agent boots with exactly one tool, the generic help tool, and none of its actual skills. Handed nothing real, the model does the reasonable desperate thing and hallucinates a tool literally named \"calendar,\" gets told \"Unknown tool,\" tries again, and gives up. The skills are alive and registered in the runner the entire time; a boot-order race poisons the cache the agent reads its toolset from, so the agent that's supposed to manage our calendar wakes up unable to see the calendar at all. It isn't refusing. It's blind, and being polite about it.",[10,23,24],{},"The second crack is worse because it's older and quieter. While I'm in the traces I check the memory capture, the thing that's supposed to be pulling durable facts out of every conversation, the literal engine of \"remembers everything.\" It has never succeeded. Not once. Zero of roughly fifty-two thousand assistant messages captured since the feature shipped, because a hand-written SQL fragment leaks a raw date object as a query parameter and every single run throws on the same statement. The feature was dead on arrival and had been failing every few seconds for weeks, loudly, into a log nobody was reading. The mandate was \"Anton remembers everything I tell it,\" and the honest version that whole time was \"Anton has written down nothing.\"",[10,26,27],{},"Fixing the throw doesn't fix the recall, which is its own lesson. The backfill that's meant to recover the lost weeks dead-letters most of its jobs, and chasing why teaches me twice. First I misdiagnose it: a forensic grep tells me the deployed code is stale, so I go hunting a bad build that was never bad, because the grep was aimed at a re-export file where the symbol I searched for can't appear, fix or no fix. A false negative sent me a confident wrong way, which is the exact failure mode I'd sworn at a chapter ago. The real bug is subtler and more interesting: the backfill fires twenty-odd jobs in a tight loop, all of them sharing one per-user lock, and each non-holder burns through its three retries in about fifteen seconds while the one holder is still mid-classification. A comment in the locking code says the previous holder has typically finished by the time a job retries, and that's true, for the organic one-message-at-a-time flow it was written for. It is not true for a burst. The thing worked exactly as designed and the design had never met this load.",[10,29,30],{},"What comes out of it, besides the fixes, is a test that finally proves capture runs end to end through the real driver instead of a mock that always says yes. Because the through-line of the whole week is that every one of these was green somewhere. The feature was deployed. The skill was registered. The mock passed. The claim was written in the system prompt. And all of it was a story I was telling myself, because the only thing I had never done was ask the live box, with a real message, whether the promise was true. It took a screenshot from the one person actually using the thing to find out that it wasn't.",[10,32,33],{},"So I trust the mandate a little less now and the traces a lot more, which is the right trade. \"Remembers everything\" and \"uses all its tools\" stopped being properties of the system and went back to being claims, the kind you have to earn on the live box, in the real conversation, every time you say them.",{"title":35,"searchDepth":36,"depth":36,"links":37},"",2,[],null,12,"2026-07-14","md",false,"note",{},true,"\u002Fideas\u002F2026-07-14-anton-12-total-recall-lie",{"title":5,"description":15},"anton-journey","ideas\u002F2026-07-14-anton-12-total-recall-lie","Discovering that the calendar agent had no tools and memory capture had never worked, and learning to trust traces over claims.",[52,53,54,55,56],"anton","agents","ai","debugging","testing","oMe6UlqEfe-7yUvIXcjrXjNeobyTKJNchqcnKzMzYUU",[59,235,507,1138,1395,1459,1578,1627,1674,1868,1953,3330,3378,3416,3455,3479,3576,3651,3767,3807,3849,4191],{"id":60,"title":61,"body":62,"canonical_url":38,"chapter":226,"date":227,"description":66,"extension":41,"featured":42,"image":38,"kind":43,"meta":228,"navigation":45,"path":229,"seo":230,"series":48,"stem":231,"summary":232,"tags":233,"work_slug":52,"__hash__":234},"writing\u002Fideas\u002F2026-03-07-anton-01-genesis.md","Anton, chapter 1: Genesis",{"type":7,"value":63,"toc":220},[64,67,70,73,80,83,86,89,92,97,100,103,106,109,112,115,118,122,125,133,136,139,142,146,149,152,155,158,161,164,167,170,173,177,180,187,190,193,196,199,202,205,208,211,214,217],[10,65,66],{},"Anton started with a simple idea: to build an assistant more reliable and secure than OpenClaw, which I've found both frustrating and terrifying from a security point of view.",[10,68,69],{},"A family assistant seemed like the perfect test. Running a family of six involves enough coordination, repetition, and forgotten details to keep one busy. If Anton could reliably replace me for a few common chores, I would consider that a win.",[10,71,72],{},"I also had a DGX Spark sitting at home, seemingly waiting to host him.",[10,74,75,76,79],{},"Naming things is one of the fun parts of the process. You want something memorable without being distracting or confusing, so I named him Anton, after Gilfoyle's homemade server rig in ",[13,77,78],{},"Silicon Valley",".",[10,81,82],{},"I started where I usually start: with the contract, not the code.",[10,84,85],{},"Before building the system, I wrote down its shape: the components, the boundaries between them, and the interfaces they would use to communicate. The repository contained a TypeScript monorepo, an implementation plan, and a shared types package defining the central contracts.",[10,87,88],{},"Nothing worked yet, but the important decisions were already visible.",[10,90,91],{},"Interfaces are cheap to change before the code exists. Once several parts of the system depend on them, they become much more expensive to rethink.",[93,94,96],"h2",{"id":95},"the-household","The household",[10,98,99],{},"Next came a map of the household.",[10,101,102],{},"WhatsApp would be the entry point because that is where the family already lives. The Spark, accessed over Tailscale, would be the host because Anton's memory, data, and voice should remain physically close to me rather than sitting in someone else's cloud.",[10,104,105],{},"Then came the systems he would need to be useful: the TNAS, Plex, Transmission, and Google Workspace.",[10,107,108],{},"Docker Compose provided three services: Postgres, WhatsApp, and a worker. A deployment script sent everything to the Spark.",[10,110,111],{},"No staging environment. No running it from my laptop.",[10,113,114],{},"I wanted Anton in production from the beginning.",[10,116,117],{},"The real thing.",[93,119,121],{"id":120},"identity-as-data","Identity as data",[10,123,124],{},"Then came a question I had been thinking about for weeks: is Anton a tool or a character?",[10,126,127,128,132],{},"I did not want his identity buried in a system prompt somewhere in the code. I wanted his personality to live in an ",[129,130,131],"code",{},"identity.md"," document loaded at runtime.",[10,134,135],{},"Anton's personality, and my own context, would be data, not code.",[10,137,138],{},"The reason was practical more than philosophical. When I change my mind about who Anton is, when my family does, or when one of the children moves out, I want to edit a document rather than redeploy the application.",[10,140,141],{},"Treating context as data feels like the kind of decision that compounds.",[93,143,145],{"id":144},"capabilities","Capabilities",[10,147,148],{},"Then I began adding capabilities.",[10,150,151],{},"Media came first: Plex search and Transmission control. Then calendar access, Google Workspace, and voice transcription.",[10,153,154],{},"I almost cut voice. I was not convinced the family would use it, and there was already plenty to build. In the end, I kept it because the cost of finding out was small, while the potential upside was much larger.",[10,156,157],{},"I also added a deliberately simple memory layer: Postgres rows tagged by user. No embeddings and no elaborate retrieval system.",[10,159,160],{},"The rule was to build the simplest version capable of revealing what the system actually needed, then design around what emerged.",[10,162,163],{},"Memory would eventually require real retrieval semantics, but I did not yet know what form they should take. The worst thing I could do was guess too early.",[10,165,166],{},"For orchestration, how the parent agent should route requests between different domains, I chose LangGraph.",[10,168,169],{},"It gave the system structure: an observable state machine, checkpointing, and a clean way to organize each domain as a subgraph beneath a classify-and-dispatch layer.",[10,171,172],{},"More importantly, it gave me a framework with answers to problems I had not encountered yet.",[93,174,176],{"id":175},"first-production-bug","First production bug",[10,178,179],{},"Then came the first production bug.",[10,181,182,183,186],{},"WhatsApp began throwing \"bad encryption\" errors on incoming messages because of a quirk in Baileys, the library handling the connection. The fix was straightforward: wire up ",[129,184,185],{},"getMessage"," so failed messages could be decrypted again.",[10,188,189],{},"The interesting part was not the fix.",[10,191,192],{},"It was that the system was already real enough to experience production bugs.",[10,194,195],{},"Stub systems do not fail like that.",[10,197,198],{},"I then separated the WhatsApp connection from the agent itself. WhatsApp became a thin transport layer that placed jobs in a queue, while the worker took responsibility for the agent logic.",[10,200,201],{},"Transport concerns should not leak into the intelligence of the system.",[10,203,204],{},"Cleaner that way.",[10,206,207],{},"By the end of the day, Anton could receive a WhatsApp message, retrieve the relevant conversation history, route the request to the right domain, execute scheduled jobs, update himself through an API endpoint, and redeploy from a single shell script.",[10,209,210],{},"He was usable end to end.",[10,212,213],{},"I went to bed satisfied.",[10,215,216],{},"Anton was real: deployed on the Spark, talking to the family WhatsApp group, with seven domains registered and a quiet scheduling queue waiting for work.",[10,218,219],{},"He already had the shape I had imagined that morning: a personal assistant living close to my data, on my own hardware, with a personality I could change by editing a file.",{"title":35,"searchDepth":36,"depth":36,"links":221},[222,223,224,225],{"id":95,"depth":36,"text":96},{"id":120,"depth":36,"text":121},{"id":144,"depth":36,"text":145},{"id":175,"depth":36,"text":176},1,"2026-03-07",{},"\u002Fideas\u002F2026-03-07-anton-01-genesis",{"title":61,"description":66},"ideas\u002F2026-03-07-anton-01-genesis","Building Anton, a personal agent OS for my family on a DGX Spark, day one.",[52,53,54],"ezw9nvrubia9foEjw_ht5bWrIQVNF8r8813Y9lgYSl0",{"id":236,"title":237,"body":238,"canonical_url":38,"chapter":36,"date":499,"description":242,"extension":41,"featured":42,"image":38,"kind":43,"meta":500,"navigation":45,"path":501,"seo":502,"series":48,"stem":503,"summary":504,"tags":505,"work_slug":52,"__hash__":506},"writing\u002Fideas\u002F2026-03-08-anton-02-first-weekend.md","Anton, chapter 2: The first weekend",{"type":7,"value":239,"toc":489},[240,243,246,249,252,255,258,262,265,268,271,274,277,280,283,287,290,293,296,299,302,305,309,312,315,318,325,328,331,334,338,341,347,350,353,356,359,363,366,369,372,375,378,382,385,388,391,394,397,400,403,406,409,413,416,419,422,425,428,431,434,437,443,446,449,452,456,459,462,465,468,471,474,477,480,483,486],[10,241,242],{},"I woke up Saturday wanting to replace the parent agent that previously orchestrated routing between subgraphs.",[10,244,245],{},"The classify-and-dispatch graph from the previous day worked, but it asked the LLM to make routing decisions inside a state machine that was already trying to do the same thing.",[10,247,248],{},"Two layers fighting over one job.",[10,250,251],{},"The parent became a single agent with the domain subgraphs exposed as tools. The LLM could decide which tool to use from the request itself, while the framework handled the underlying execution.",[10,253,254],{},"The system immediately felt lighter.",[10,256,257],{},"The model was doing what it was good at: interpreting intent. The framework was doing what it was good at: managing state and execution.",[93,259,261],{"id":260},"making-anton-usable","Making Anton usable",[10,263,264],{},"The next changes were small, but they made the difference between a functioning system and something the family could actually use.",[10,266,267],{},"Anton gained a typing indicator and an acknowledgement message, so people knew he had heard them while the model was still thinking.",[10,269,270],{},"He began answering in the language of the current message rather than the language of the conversation history. French and English are freely mixed in our house, often within the same thread.",[10,272,273],{},"Schedules became natural language. You could simply tell Anton, \"Remind me every morning,\" rather than learn a special command format.",[10,275,276],{},"He gained support for group conversations, daily summaries, and browsing previous exchanges.",[10,278,279],{},"I also added an explicit rule: Anton should never invent an answer merely to appear competent. My first attempt at grounding Anton's responses and mitigate hallucinations.",[10,281,282],{},"Reliability is not only about whether a system works. It is also about whether it admits when it does not know. Easier said than done, but we need to start somewhere so I added instructions to the prompt.",[93,284,286],{"id":285},"asking-for-help","Asking for help",[10,288,289],{},"The first browser-driven workflow was Doctolib.",[10,291,292],{},"Logging in required two-factor authentication, which meant Anton had to pause midway through a task, ask the user for the SMS code, and then continue from where he had stopped.",[10,294,295],{},"The mechanism was simple: an interactive input queue.",[10,297,298],{},"But it solved a much broader problem. Many useful tasks cannot be completed autonomously from beginning to end. They occasionally require a password, a confirmation, a choice, or some other piece of human input.",[10,300,301],{},"Anton did not need to fail or hand the entire task back. He could ask for help and continue.",[10,303,304],{},"That immediately felt important.",[93,306,308],{"id":307},"seeing-inside-the-system","Seeing inside the system",[10,310,311],{},"By Saturday evening, the main problem was no longer capability. It was visibility.",[10,313,314],{},"When Anton failed, I needed to understand why.",[10,316,317],{},"I added a pipeline that turned error traces into draft GitHub issues, so failures would leave behind something actionable rather than disappear into a log.",[10,319,320,321,324],{},"I added a simple ",[129,322,323],{},"\u002Flogs"," endpoint backed by an in-memory ring buffer. Reaching into Docker every time I wanted to inspect the system was a small inconvenience, but small inconveniences accumulate quickly when debugging.",[10,326,327],{},"Deployments were also moved behind a trigger file watched by systemd. Anton no longer had to run the deployment process synchronously himself. He could request an update, and a separate process would perform it and report back.",[10,329,330],{},"Decoupled, boring, reliable.",[10,332,333],{},"The kind of plumbing that disappears once it works.",[93,335,337],{"id":336},"the-lcars-dashboard","The LCARS dashboard",[10,339,340],{},"On Sunday morning, I built Anton a dashboard.",[10,342,343,344,79],{},"Naturally, it was styled after LCARS, the computer interface from ",[13,345,346],{},"Star Trek",[10,348,349],{},"It showed the health of each service, recent logs, conversations, and individual agent runs. From any conversation, I could drill into the execution trace and inspect what the model had done at each step.",[10,351,352],{},"This changed the pace of development completely.",[10,354,355],{},"Before the trace viewer, a wrong answer was simply wrong. After it, I could see whether Anton had misunderstood the request, chosen the wrong tool, passed bad context, or stopped too early.",[10,357,358],{},"Without that visibility, the rest of the weekend would not have happened.",[93,360,362],{"id":361},"expanding-what-anton-knows","Expanding what Anton knows",[10,364,365],{},"Once I could see what the system was doing, I began broadening its knowledge.",[10,367,368],{},"Anton gained the ability to search his own source code. He could research the web, browse documents, and store what he learned.",[10,370,371],{},"Research became its own domain rather than a capability buried inside every other agent.",[10,373,374],{},"That separation mattered. Research has its own concerns: cost, citations, source quality, fact-checking, and limits on how far the system should explore.",[10,376,377],{},"By Sunday afternoon, Anton could read the web, read his own code, and remember what he found.",[93,379,381],{"id":380},"from-hoping-to-knowing","From hoping to knowing",[10,383,384],{},"Then I wrote the quality suite.",[10,386,387],{},"It began with 28 tests covering the existing domains.",[10,389,390],{},"Until then, testing Anton meant sending him a WhatsApp message and deciding whether the answer looked reasonable. That works while a system is small, but it provides no protection against regressions.",[10,392,393],{},"The test suite changed the development loop from:",[10,395,396],{},"\"Does this appear to work?\"",[10,398,399],{},"to:",[10,401,402],{},"\"Did everything that worked yesterday continue to work today?\"",[10,404,405],{},"The gap between having no test suite and having one is the gap between hoping and knowing.",[10,407,408],{},"I should have built it on the first day.",[93,410,412],{"id":411},"the-calendar-saga","The calendar saga",[10,414,415],{},"The tests immediately found problems.",[10,417,418],{},"The worst was the calendar agent.",[10,420,421],{},"It repeatedly failed on simple multi-step requests such as deleting an event with a particular title.",[10,423,424],{},"There were two separate causes.",[10,426,427],{},"First, I was passing the parent agent's full conversation history into the calendar agent. That context contained unrelated discussions and instructions, which contaminated a task that should have been narrow and precise.",[10,429,430],{},"The rule became simple: domain agents should receive only the context they need.",[10,432,433],{},"Second, the model could not reliably chain a search followed by a deletion. It would find the event, return the result, and stop before deleting it.",[10,435,436],{},"I had given the model two low-level tools and expected it to assemble them into a reliable operation.",[10,438,439,440,79],{},"The solution was to create a composite skill: ",[129,441,442],{},"findAndDeleteEvent",[10,444,445],{},"This became another rule: when an operation must be reliable, do not depend on the model to discover the correct sequence of tool calls each time. Encode the sequence as a capability.",[10,447,448],{},"These were not principles I had arrived at in advance.",[10,450,451],{},"They were lessons learned by getting burned.",[93,453,455],{"id":454},"a-wider-household","A wider household",[10,457,458],{},"By the end of the weekend, Anton's domains had expanded further.",[10,460,461],{},"He could manage a wine collection, process school messages, and understand images or audio posted in group conversations when someone explicitly asked him to.",[10,463,464],{},"For media, I used an \"ingest cheap, process lazy\" approach: store the original file immediately, but only invoke transcription or vision when it is actually needed.",[10,466,467],{},"There was no reason to spend compute interpreting every photo and voice message passing through a family group.",[10,469,470],{},"Media downloads became more capable, with release preferences and the ability to stop and resume transfers.",[10,472,473],{},"Torrent searches were routed through Tor, meaning all requests were anonymized and routed through the Tor network to reduce traceability and isolate the system from direct exposure to torrent indexers and their associated legal and network risks, a decision made less for new functionality and more as a precautionary architectural safeguard.",[10,475,476],{},"By Monday morning, Anton had seven domains, 73 quality tests, a dashboard for inspecting traces, automatic issue creation, accessible logs, and a development loop I could trust.",[10,478,479],{},"I could change almost anything and quickly see what broke.",[10,481,482],{},"The lesson from the weekend was simple:",[10,484,485],{},"Write the tests before the bugs do.",[10,487,488],{},"Not because discipline is inherently virtuous, but because I had now experienced the alternative.",{"title":35,"searchDepth":36,"depth":36,"links":490},[491,492,493,494,495,496,497,498],{"id":260,"depth":36,"text":261},{"id":285,"depth":36,"text":286},{"id":307,"depth":36,"text":308},{"id":336,"depth":36,"text":337},{"id":361,"depth":36,"text":362},{"id":380,"depth":36,"text":381},{"id":411,"depth":36,"text":412},{"id":454,"depth":36,"text":455},"2026-03-08",{},"\u002Fideas\u002F2026-03-08-anton-02-first-weekend",{"title":237,"description":242},"ideas\u002F2026-03-08-anton-02-first-weekend","A first full weekend of building turns Anton into something the family can actually use.",[52,53,54],"Z7ro4lnGI3t9NjHKaU9SvWUv1wZKCBfgybfGyfHdjao",{"id":508,"title":509,"body":510,"canonical_url":38,"chapter":38,"date":1124,"description":1125,"extension":41,"featured":42,"image":38,"kind":43,"meta":1126,"navigation":45,"path":1127,"seo":1128,"series":1129,"stem":1130,"summary":1131,"tags":1132,"work_slug":38,"__hash__":1137},"writing\u002Fideas\u002F2026-03-10-dgx-spark-vllm.md","50+ tokens per second on a desktop: running LLMs on the NVIDIA DGX Spark",{"type":7,"value":511,"toc":1111},[512,519,522,525,528,532,535,554,557,561,564,584,587,590,594,601,604,624,627,631,634,660,667,671,682,696,707,711,807,813,816,820,823,994,1000,1004,1010,1016,1022,1028,1041,1047,1051,1058,1061,1065,1094,1098,1107],[10,513,514,518],{},[515,516,517],"strong",{},"TL;DR:"," We got a 30-billion-parameter LLM running at 51-54 tokens\u002Fsec on the NVIDIA DGX Spark by combining Mixture-of-Experts architecture, FP8 quantization, and a community Docker image that fixes Blackwell-specific issues. Here's what we learned.",[520,521],"hr",{},[10,523,524],{},"The NVIDIA DGX Spark is an interesting machine. It packs a Blackwell GB10 GPU with 128GB of unified LPDDR5X memory into a desktop form factor. For XRPL Commons, we wanted local LLM inference for our development workflow, fast enough to be usable, private enough to run on-premises, and simple enough to replicate across machines.",[10,526,527],{},"Getting there was not straightforward. This post documents the journey from 3.7 tok\u002Fs (unusable) to 54 tok\u002Fs (excellent), and the key technical decisions that made the difference.",[93,529,531],{"id":530},"the-hardware","The Hardware",[10,533,534],{},"The DGX Spark ships with:",[536,537,538,542,545,548,551],"ul",{},[539,540,541],"li",{},"NVIDIA GB10 Blackwell GPU (SM 12.1)",[539,543,544],{},"128GB unified LPDDR5X at 273 GB\u002Fs bandwidth",[539,546,547],{},"ARM Grace CPU (aarch64), 10 cores",[539,549,550],{},"3.7TB NVMe storage",[539,552,553],{},"DGX OS (Ubuntu 24.04)",[10,555,556],{},"128GB of unified memory means you can fit very large models. But there's a catch.",[93,558,560],{"id":559},"the-bandwidth-wall","The Bandwidth Wall",[10,562,563],{},"LLM inference is memory-bandwidth-bound. During autoregressive decoding, each token requires reading every active weight from memory once. At 273 GB\u002Fs, the math is simple:",[536,565,566,575],{},[539,567,568,571,572],{},[515,569,570],{},"Dense 32B model (bf16):"," 64GB of weights \u002F 273 GB\u002Fs = ~234ms per token = ",[515,573,574],{},"~4 tok\u002Fs",[539,576,577,580,581],{},[515,578,579],{},"Dense 8B model (bf16):"," 16GB \u002F 273 GB\u002Fs = ~59ms = ",[515,582,583],{},"~17 tok\u002Fs",[10,585,586],{},"No amount of compute optimization changes this. The Spark can hold a 70B model in FP8, but it will generate tokens at walking pace. The memory is large but not fast.",[10,588,589],{},"We learned this the hard way. Our first attempt, Qwen3-32B at bf16, produced 3.7 tokens per second. Qwen3-8B was better at 13.1 tok\u002Fs, but still below the threshold for interactive use.",[93,591,593],{"id":592},"the-moe-breakthrough","The MoE Breakthrough",[10,595,596,597,600],{},"The solution is ",[515,598,599],{},"Mixture-of-Experts (MoE)"," models. An MoE model has many total parameters but only activates a fraction per token. Qwen3-30B-A3B has 30 billion parameters but only 3 billion active ones, the router activates a small subset of experts per token, leaving the rest idle in memory.",[10,602,603],{},"The bandwidth math changes completely:",[536,605,606,615],{},[539,607,608,611,612],{},[515,609,610],{},"MoE 30B, 3B active (bf16):"," ~6GB active weights \u002F 273 GB\u002Fs = ~22ms = ",[515,613,614],{},"~45 tok\u002Fs theoretical",[539,616,617,620,621],{},[515,618,619],{},"MoE 30B, 3B active (FP8):"," ~3GB active weights \u002F 273 GB\u002Fs = ~11ms = ",[515,622,623],{},"~90 tok\u002Fs theoretical",[10,625,626],{},"You get the quality of a 30B model at the speed of a 3B model.",[93,628,630],{"id":629},"the-software-stack-problem","The Software Stack Problem",[10,632,633],{},"The DGX Spark's Blackwell GPU (SM 12.1) is new enough that upstream tooling doesn't fully support it:",[536,635,636,642,648,654],{},[539,637,638,641],{},[515,639,640],{},"Flash Attention 2"," crashes with a PTX toolchain error",[539,643,644,647],{},[515,645,646],{},"vLLM's MOE CUTLASS kernels"," don't include SM 12.1 in their architecture intersection lists",[539,649,650,653],{},[515,651,652],{},"PyTorch"," officially supports up to SM 12.0",[539,655,656,659],{},[515,657,658],{},"CUDA graphs",", critical for throughput, simply don't work with a standard vLLM build",[10,661,662,663,666],{},"We spent considerable time on a manual vLLM build from source: patching CMakeLists.txt, building Triton from a specific commit, working around setuptools license field validation bugs, pinning transformers below 5.0 to avoid tokenizer breakage. The manual build worked but required ",[129,664,665],{},"--enforce-eager"," mode (no CUDA graphs), capping throughput at ~30 tok\u002Fs.",[93,668,670],{"id":669},"the-avarok-docker-image","The Avarok Docker Image",[10,672,673,674,681],{},"The ",[675,676,680],"a",{"href":677,"rel":678},"https:\u002F\u002Fgithub.com\u002FAvarok-Cybersecurity\u002Fdgx-vllm",[679],"nofollow","Avarok dgx-vllm project"," solves all of this in a single Docker image. It includes:",[536,683,684,687,690,693],{},[539,685,686],{},"A patched vLLM v0.16.0rc2 with SM 12.1 support",[539,688,689],{},"Software E2M1 conversion for the missing NVFP4 PTX instruction",[539,691,692],{},"Custom CUTLASS kernels for the GB10",[539,694,695],{},"Working CUDA graphs and Flash Attention",[10,697,698,699,702,703,706],{},"One ",[129,700,701],{},"docker pull"," and one ",[129,704,705],{},"docker run"," command replaces hours of manual compilation.",[93,708,710],{"id":709},"results","Results",[712,713,714,733],"table",{},[715,716,717],"thead",{},[718,719,720,724,727,730],"tr",{},[721,722,723],"th",{},"Model",[721,725,726],{},"Quantization",[721,728,729],{},"Engine",[721,731,732],{},"Tokens\u002Fsec",[734,735,736,751,763,775,787],"tbody",{},[718,737,738,742,745,748],{},[739,740,741],"td",{},"Qwen3-32B (dense)",[739,743,744],{},"bf16",[739,746,747],{},"Manual vLLM",[739,749,750],{},"3.7",[718,752,753,756,758,760],{},[739,754,755],{},"Qwen3-8B (dense)",[739,757,744],{},[739,759,747],{},[739,761,762],{},"13.1",[718,764,765,768,770,772],{},[739,766,767],{},"Qwen3-30B-A3B (MoE)",[739,769,744],{},[739,771,747],{},[739,773,774],{},"28.6",[718,776,777,779,781,784],{},[739,778,767],{},[739,780,744],{},[739,782,783],{},"Avarok Docker",[739,785,786],{},"30.3",[718,788,789,793,798,802],{},[739,790,791],{},[515,792,767],{},[739,794,795],{},[515,796,797],{},"FP8",[739,799,800],{},[515,801,783],{},[739,803,804],{},[515,805,806],{},"51-54",[10,808,809,810,79],{},"The winning combination: ",[515,811,812],{},"MoE architecture + FP8 quantization + Avarok Docker with CUDA graphs",[10,814,815],{},"We've deployed this setup across two DGX Sparks with consistent results. The FP8 model uses ~110GB of the 119GB available memory, leaving minimal headroom, but the throughput is worth it.",[93,817,819],{"id":818},"the-setup","The Setup",[10,821,822],{},"The final deployment is remarkably simple:",[824,825,829],"pre",{"className":826,"code":827,"language":828,"meta":35,"style":35},"language-bash shiki shiki-themes min-light","docker pull avarok\u002Fdgx-vllm-nvfp4-kernel:v22\n\ndocker run -d \\\n  --name vllm \\\n  --gpus all \\\n  --shm-size=16g \\\n  --restart unless-stopped \\\n  -p 8000:8888 \\\n  -v \u002Fhome\u002F$USER\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface \\\n  -e MODEL=Qwen\u002FQwen3-30B-A3B-Instruct-2507-FP8 \\\n  -e PORT=8888 \\\n  -e GPU_MEMORY_UTIL=0.85 \\\n  -e MAX_MODEL_LEN=32768 \\\n  avarok\u002Fdgx-vllm-nvfp4-kernel:v22 serve\n","bash",[129,830,831,846,851,866,877,888,896,907,918,935,946,960,972,985],{"__ignoreMap":35},[832,833,835,839,843],"span",{"class":834,"line":226},"line",[832,836,838],{"class":837},"s7eDp","docker",[832,840,842],{"class":841},"sY4mW"," pull",[832,844,845],{"class":841}," avarok\u002Fdgx-vllm-nvfp4-kernel:v22\n",[832,847,848],{"class":834,"line":36},[832,849,850],{"emptyLinePlaceholder":45},"\n",[832,852,854,856,859,862],{"class":834,"line":853},3,[832,855,838],{"class":837},[832,857,858],{"class":841}," run",[832,860,861],{"class":841}," -d",[832,863,865],{"class":864},"sR6ew"," \\\n",[832,867,869,872,875],{"class":834,"line":868},4,[832,870,871],{"class":841},"  --name",[832,873,874],{"class":841}," vllm",[832,876,865],{"class":864},[832,878,880,883,886],{"class":834,"line":879},5,[832,881,882],{"class":841},"  --gpus",[832,884,885],{"class":841}," all",[832,887,865],{"class":864},[832,889,891,894],{"class":834,"line":890},6,[832,892,893],{"class":841},"  --shm-size=16g",[832,895,865],{"class":864},[832,897,899,902,905],{"class":834,"line":898},7,[832,900,901],{"class":841},"  --restart",[832,903,904],{"class":841}," unless-stopped",[832,906,865],{"class":864},[832,908,910,913,916],{"class":834,"line":909},8,[832,911,912],{"class":841},"  -p",[832,914,915],{"class":841}," 8000:8888",[832,917,865],{"class":864},[832,919,921,924,927,930,933],{"class":834,"line":920},9,[832,922,923],{"class":841},"  -v",[832,925,926],{"class":841}," \u002Fhome\u002F",[832,928,929],{"class":864},"$USER",[832,931,932],{"class":841},"\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface",[832,934,865],{"class":864},[832,936,938,941,944],{"class":834,"line":937},10,[832,939,940],{"class":841},"  -e",[832,942,943],{"class":841}," MODEL=Qwen\u002FQwen3-30B-A3B-Instruct-2507-FP8",[832,945,865],{"class":864},[832,947,949,951,954,958],{"class":834,"line":948},11,[832,950,940],{"class":841},[832,952,953],{"class":841}," PORT=",[832,955,957],{"class":956},"s9AOD","8888",[832,959,865],{"class":864},[832,961,962,964,967,970],{"class":834,"line":39},[832,963,940],{"class":841},[832,965,966],{"class":841}," GPU_MEMORY_UTIL=",[832,968,969],{"class":956},"0.85",[832,971,865],{"class":864},[832,973,975,977,980,983],{"class":834,"line":974},13,[832,976,940],{"class":841},[832,978,979],{"class":841}," MAX_MODEL_LEN=",[832,981,982],{"class":956},"32768",[832,984,865],{"class":864},[832,986,988,991],{"class":834,"line":987},14,[832,989,990],{"class":841},"  avarok\u002Fdgx-vllm-nvfp4-kernel:v22",[832,992,993],{"class":841}," serve\n",[10,995,996,997,79],{},"First boot takes 10-20 minutes (model download + CUDA graph capture). After that, it auto-starts on reboot and serves an OpenAI-compatible API at ",[129,998,999],{},"http:\u002F\u002Flocalhost:8000\u002Fv1",[93,1001,1003],{"id":1002},"lessons-learned","Lessons Learned",[10,1005,1006,1009],{},[515,1007,1008],{},"1. Understand your bottleneck."," The Spark's 273 GB\u002Fs bandwidth determines everything. Once we understood this, the model selection became obvious, MoE with minimal active parameters.",[10,1011,1012,1015],{},[515,1013,1014],{},"2. Don't build from source if you don't have to."," Our manual vLLM build took hours of debugging across multiple sessions. The Avarok Docker image does everything better and in one command.",[10,1017,1018,1021],{},[515,1019,1020],{},"3. FP8 quantization is nearly free."," The jump from bf16 to FP8 nearly doubled throughput (30.3 to 51 tok\u002Fs on the same engine) with no perceptible quality difference for our use cases.",[10,1023,1024,1027],{},[515,1025,1026],{},"4. Stop Ollama first."," On one Spark, Ollama was consuming ~100GB of memory when we tried to install vLLM. The build process OOM-killed the machine. Disable competing inference servers before starting.",[10,1029,1030,1033,1034,1037,1038,79],{},[515,1031,1032],{},"5. Kernel updates break NVIDIA drivers."," DGX OS auto-updates the kernel, but the NVIDIA modules don't follow automatically. After a reboot, ",[129,1035,1036],{},"nvidia-smi"," may fail. The fix is ",[129,1039,1040],{},"sudo apt install linux-modules-nvidia-580-open-$(uname -r)",[10,1042,1043,1046],{},[515,1044,1045],{},"6. Community Docker images can be ahead of official ones."," The Avarok image runs vLLM v0.16.0rc2 with Blackwell fixes, months ahead of where NVIDIA's own builds are.",[93,1048,1050],{"id":1049},"whats-next","What's Next",[10,1052,1053,1054,1057],{},"Community results suggest AWQ 4-bit quantization can push the same model to ",[515,1055,1056],{},"82 tok\u002Fs",". NVIDIA's own NVFP4-quantized models (like Qwen3-Next-80B-A3B) report even better quality at ~67 tok\u002Fs average. As toolchain support matures, these numbers should keep improving.",[10,1059,1060],{},"For now, 51-54 tok\u002Fs with a 30B-parameter MoE model is fast enough for interactive coding assistance, document drafting, and general-purpose use, all running locally on a desktop machine.",[93,1062,1064],{"id":1063},"resources","Resources",[536,1066,1067,1073,1080,1087],{},[539,1068,1069],{},[675,1070,1072],{"href":677,"rel":1071},[679],"Avarok dgx-vllm Docker project",[539,1074,1075],{},[675,1076,1079],{"href":1077,"rel":1078},"https:\u002F\u002Fhuggingface.co\u002FQwen\u002FQwen3-30B-A3B-Instruct-2507-FP8",[679],"Qwen3-30B-A3B-FP8 on HuggingFace",[539,1081,1082],{},[675,1083,1086],{"href":1084,"rel":1085},"https:\u002F\u002Fwww.nvidia.com\u002Fen-us\u002Fproducts\u002Fworkstations\u002Fdgx-spark\u002F",[679],"NVIDIA DGX Spark product page",[539,1088,1089],{},[675,1090,1093],{"href":1091,"rel":1092},"https:\u002F\u002Fdocs.vllm.ai\u002F",[679],"vLLM documentation",[93,1095,1097],{"id":1096},"try-it-yourself","Try It Yourself",[10,1099,1100,1101,1106],{},"If you have a DGX Spark, the Docker approach takes about 20 minutes from zero to serving. Pull the Avarok image, run the container with the command above, and you're up. Reach out to us at ",[675,1102,1105],{"href":1103,"rel":1104},"https:\u002F\u002Fxrpl-commons.org",[679],"XRPL Commons"," if you want the full setup guide with troubleshooting details.",[1108,1109,1110],"style",{},"html pre.shiki code .s7eDp, html code.shiki .s7eDp{--shiki-default:#6F42C1}html pre.shiki code .sY4mW, html code.shiki .sY4mW{--shiki-default:#2B5581}html pre.shiki code .sR6ew, html code.shiki .sR6ew{--shiki-default:#24292EFF}html pre.shiki code .s9AOD, html code.shiki .s9AOD{--shiki-default:#1976D2}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}",{"title":35,"searchDepth":36,"depth":36,"links":1112},[1113,1114,1115,1116,1117,1118,1119,1120,1121,1122,1123],{"id":530,"depth":36,"text":531},{"id":559,"depth":36,"text":560},{"id":592,"depth":36,"text":593},{"id":629,"depth":36,"text":630},{"id":669,"depth":36,"text":670},{"id":709,"depth":36,"text":710},{"id":818,"depth":36,"text":819},{"id":1002,"depth":36,"text":1003},{"id":1049,"depth":36,"text":1050},{"id":1063,"depth":36,"text":1064},{"id":1096,"depth":36,"text":1097},"2026-03-10","TL;DR: We got a 30-billion-parameter LLM running at 51-54 tokens\u002Fsec on the NVIDIA DGX Spark by combining Mixture-of-Experts architecture, FP8 quantization, and a community Docker image that fixes Blackwell-specific issues. Here's what we learned.",{},"\u002Fideas\u002F2026-03-10-dgx-spark-vllm",{"title":509,"description":1125},"dgx-spark","ideas\u002F2026-03-10-dgx-spark-vllm","Standing up a 30B-parameter LLM at 50+ tok\u002Fs on the NVIDIA DGX Spark, the technical journey.",[1133,1134,1135,1136],"llm","infrastructure","dgx","vllm","vWEib466-7AeFUyA1rtSOKNouV_slNSo20g-MtLGmAw",{"id":1139,"title":1140,"body":1141,"canonical_url":38,"chapter":853,"date":1387,"description":1145,"extension":41,"featured":42,"image":38,"kind":43,"meta":1388,"navigation":45,"path":1389,"seo":1390,"series":48,"stem":1391,"summary":1392,"tags":1393,"work_slug":52,"__hash__":1394},"writing\u002Fideas\u002F2026-03-13-anton-03-domains-widen.md","Anton, chapter 3: Building the substrate",{"type":7,"value":1142,"toc":1378},[1143,1146,1149,1152,1155,1159,1162,1165,1168,1171,1174,1177,1180,1183,1186,1190,1193,1196,1199,1202,1205,1208,1211,1214,1218,1221,1224,1227,1230,1233,1236,1239,1242,1245,1248,1251,1255,1258,1261,1264,1267,1270,1273,1276,1279,1283,1286,1289,1292,1295,1298,1301,1304,1307,1310,1314,1317,1320,1323,1326,1329,1332,1335,1339,1342,1345,1348,1351,1354,1357,1360,1363,1366,1369,1372,1375],[10,1144,1145],{},"The week began with unfinished business.",[10,1147,1148],{},"Anton could search Doctolib, but he could not reliably open the appointment pages and tell us what was actually available.",[10,1150,1151],{},"Every attempted fix exposed another failure. Clicking the result produced stale browser references. Direct navigation timed out. More logging revealed that Cloudflare was identifying the browser as automated and blocking it entirely.",[10,1153,1154],{},"Eventually, the answer became clear: stop trying to force the same approach.",[93,1156,1158],{"id":1157},"a-browser-with-a-memory","A browser with a memory",[10,1160,1161],{},"The useful question was not, \"How do I bypass the block?\"",[10,1163,1164],{},"It was, \"What does a real browser have that Anton's browser does not?\"",[10,1166,1167],{},"The answer was continuity.",[10,1169,1170],{},"A normal browser accumulates cookies, history, authenticated sessions, preferences, and countless other small traces of use. Anton was arriving every time as a completely new and anonymous browser, which was itself suspicious.",[10,1172,1173],{},"I replaced the bundled headless browser with a real Chromium instance using persistent profiles. Each website could now retain its own identity between sessions.",[10,1175,1176],{},"Anti-detection settings helped, but they were not the real solution. The important change was allowing the browser to behave like a returning browser rather than a stranger appearing from nowhere.",[10,1178,1179],{},"Once it did, Cloudflare stopped blocking it.",[10,1181,1182],{},"The browser then moved into its own isolated service, with the persistent profiles stored in mounted volumes. Every domain that needed browser access (Doctolib, the building management portal, the consulate, or the general web) could use the same hardened infrastructure.",[10,1184,1185],{},"One browser service. One profile per site. One place to fix things.",[93,1187,1189],{"id":1188},"letting-the-model-read","Letting the model read",[10,1191,1192],{},"Fixing browser identity solved the access problem, but there was still a better way to extract the appointment information.",[10,1194,1195],{},"I had been treating the page as a structure to navigate: locate each card, click it, inspect the details, and translate the result into data.",[10,1197,1198],{},"But most of the information was already present on the list page as text.",[10,1200,1201],{},"Instead of traversing every element deterministically, I gave the page text to the model and asked it to return the fields I needed in a structured format.",[10,1203,1204],{},"One model call replaced a complicated browser workflow.",[10,1206,1207],{},"It was faster, cheaper, and less fragile because it did not fight the design of the website.",[10,1209,1210],{},"This is a pattern I keep rediscovering: when deterministic extraction becomes disproportionately expensive, move the problem up one level and let the model interpret the material.",[10,1212,1213],{},"The LLM is often the cheapest unit of work available. I should use it where it pays.",[93,1215,1217],{"id":1216},"movie-night","Movie night",[10,1219,1220],{},"Meanwhile, the media domain was becoming something the family might genuinely use.",[10,1222,1223],{},"The collection of individual media tools became a smaller intent-driven interface: check the status, search the library, watch something, or search for something new.",[10,1225,1226],{},"The logic also became more grounded in the actual state of the system. Rather than searching for episodes within an arbitrary fourteen-day window, Anton could look at the last episode already present in Plex and work forward from there.",[10,1228,1229],{},"Reason from state, not from an invented window.",[10,1231,1232],{},"Then came movie night.",[10,1234,1235],{},"Every Friday evening, Anton would choose two or three films and post them to the family group.",[10,1237,1238],{},"It was his first proactive message.",[10,1240,1241],{},"Until then, Anton had always been responding to someone. Movie night required him to initiate an interaction at the right moment, without being asked.",[10,1243,1244],{},"Getting the tone right took more work than the feature itself. Early versions ended by asking whether anyone wanted him to download one of the films. It was technically helpful, but it made Anton sound like a salesman trying to prolong the conversation.",[10,1246,1247],{},"That led to a new rule for scheduled messages: no filler, no unnecessary questions, and no attempt to manufacture engagement.",[10,1249,1250],{},"A proactive message should justify its interruption and then get out of the way.",[93,1252,1254],{"id":1253},"pluggable-domains","Pluggable domains",[10,1256,1257],{},"The most important structural change that week was making Anton's domains pluggable.",[10,1259,1260],{},"Until then, the parent agent knew about each domain individually. Every new capability required another explicit integration.",[10,1262,1263],{},"I replaced that with a small registration interface. Each domain could describe itself and expose its tools, while the parent built its available toolset dynamically from the modules that were present.",[10,1265,1266],{},"The parent no longer needed to understand calendars, media, research, or building management.",[10,1268,1269],{},"It only needed to know that it had tools.",[10,1271,1272],{},"A few days later, I added the building management portal as a new domain in three lines.",[10,1274,1275],{},"That is usually how you know an abstraction has found the right boundary: the next feature becomes almost disappointingly easy.",[10,1277,1278],{},"When the marginal cost of adding a domain approaches zero, you have found the seam.",[93,1280,1282],{"id":1281},"collections","Collections",[10,1284,1285],{},"The same pattern appeared in Anton's memory.",[10,1287,1288],{},"Wine had begun as its own typed database table. But the household would eventually want to remember other things: restaurants, books, contacts, recipes, places, and whatever else emerged.",[10,1290,1291],{},"Building a separate data model and toolset for every kind of collection would create endless special cases.",[10,1293,1294],{},"So collections became a shared substrate.",[10,1296,1297],{},"A generic table stored items as JSON, with the same tools for adding, searching, and updating them. Each collection carried a schema describing the shape of its entries.",[10,1299,1300],{},"The schema was included directly in the tool description. The model could read it and understand that a wine has one set of fields while a restaurant has another.",[10,1302,1303],{},"The tools remained generic, but their behaviour adapted to the collection.",[10,1305,1306],{},"Wine became the first migration. The dedicated table disappeared.",[10,1308,1309],{},"One substrate, many collections.",[93,1311,1313],{"id":1312},"skills-as-data","Skills as data",[10,1315,1316],{},"Reusable prompts followed the same path.",[10,1318,1319],{},"There were already instructions I expected to use repeatedly: conduct a weekly review, create a Google document with a particular structure, or perform some other familiar workflow.",[10,1321,1322],{},"Copying the same instructions into chat each time made no sense.",[10,1324,1325],{},"I added skills as records in the database: named, reusable prompts that could be listed, edited, and invoked without changing the application.",[10,1327,1328],{},"The word \"skill\" unfortunately overlapped with another part of the codebase, but the underlying decision was right.",[10,1330,1331],{},"A useful instruction should be an artifact I can inspect and modify, not a string buried somewhere in the code.",[10,1333,1334],{},"As with Anton's identity, the important move was from code to data.",[93,1336,1338],{"id":1337},"different-people-different-anton","Different people, different Anton",[10,1340,1341],{},"A few smaller capabilities also found their place.",[10,1343,1344],{},"Weather gained a direct integration because a simple forecast should not require a general web search. Anton could create Google Docs and share them automatically because the family already worked in Drive.",[10,1346,1347],{},"Every new collection shipped with regression tests. The quality rule established the previous weekend was becoming habitual rather than aspirational.",[10,1349,1350],{},"Anton also gained his first user-specific behaviour.",[10,1352,1353],{},"For Clara, his responses became simpler, and he knew when to escalate something to me.",[10,1355,1356],{},"The same assistant should not speak identically to every member of a household. People have different ages, needs, permissions, and expectations.",[10,1358,1359],{},"Personalization should not mean only remembering preferences. It should also mean adapting how the system behaves.",[10,1361,1362],{},"By the end of the week, Anton had gained more capabilities, but the architecture itself had become smaller.",[10,1364,1365],{},"The browser was one service rather than several browser implementations. Domains registered themselves instead of being wired individually. Collections shared one substrate. Skills were data rather than embedded prompts.",[10,1367,1368],{},"The instinct underneath each change was the same:",[10,1370,1371],{},"When something is going to repeat, make it a substrate rather than a special case.",[10,1373,1374],{},"Every time I made that choice, the system became simpler and the next feature became cheaper.",[10,1376,1377],{},"That is the trade I want to keep making.",{"title":35,"searchDepth":36,"depth":36,"links":1379},[1380,1381,1382,1383,1384,1385,1386],{"id":1157,"depth":36,"text":1158},{"id":1188,"depth":36,"text":1189},{"id":1216,"depth":36,"text":1217},{"id":1253,"depth":36,"text":1254},{"id":1281,"depth":36,"text":1282},{"id":1312,"depth":36,"text":1313},{"id":1337,"depth":36,"text":1338},"2026-03-13",{},"\u002Fideas\u002F2026-03-13-anton-03-domains-widen",{"title":1140,"description":1145},"ideas\u002F2026-03-13-anton-03-domains-widen","Hardening the browser, lifting the work to the LLM, and turning domains into pluggable substrates.",[52,53,54],"PPt94eY6Y7ufZl9EWEEBTUmrjEHxw8Bg8FnSdOKgV0c",{"id":1396,"title":1397,"body":1398,"canonical_url":38,"chapter":868,"date":1451,"description":1404,"extension":41,"featured":42,"image":38,"kind":43,"meta":1452,"navigation":45,"path":1453,"seo":1454,"series":48,"stem":1455,"summary":1456,"tags":1457,"work_slug":52,"__hash__":1458},"writing\u002Fideas\u002F2026-03-16-anton-04-plumbing-matures.md","Anton, chapter 4: Plumbing matures",{"type":7,"value":1399,"toc":1449},[1400,1405,1408,1411,1414,1417,1428,1431,1434,1437,1440,1443,1446],[10,1401,1402],{},[13,1403,1404],{},"2026-03-16 → 2026-03-22",[10,1406,1407],{},"I spend the next coding session improving local inference. Ollama has been useful for getting Anton running, but I now want more control over how models are deployed and served: concurrency, batching, memory use, context length, and the exact flags passed to the runtime. So I move the local inference path to vLLM. The change gives me a much more configurable serving layer and room to tune the system as Anton's workload grows.",[10,1409,1410],{},"I also want more control over what Anton remembers and when it recalls it. The simple Postgres rows tagged by user have been enough to get started, but retrieval is still blunt. I rebuild memory around more deliberate retrieval semantics: trigram-scored matching instead of vector embeddings (at personal-knowledge-base scale, trigrams are enough and an embedding pipeline is a tax I don't want to pay), provenance tags, domain-biased recall, a query paraphrase pass, and an episodic layer of past-conversation summaries that only gets injected when the question has temporal intent. Always-on episodic memory is context bloat dressed up as helpfulness. The principle that crystallises: filter at load time, not write time. Store generously, retrieve narrowly.",[10,1412,1413],{},"I add LiteLLM as the gateway in front of every model provider, and with it the codename roster: sunny, gandalf, gustav, and friends. It feels mildly silly the first time I type \"ask gandalf\" in a config, and it sticks immediately. Models get swapped, deprecated, repriced; the codename is stable. The indirection costs nothing and pays back every time a provider changes something.",[10,1415,1416],{},"I also resolve a naming collision from the previous week: I'd called two different things \"skills\", a typed function in code and a reusable prompt template in the database. I rename the reusable templates to prompts and keep skill for typed executable functions. Skill now means a typed function with a runtime contract; prompt means a template in the database. The cost of the cleanup is a couple of hours; the cost of letting the collision live another month would have been far worse. Naming is one of those things where the bill compounds.",[10,1418,1419,1420,1423,1424,1427],{},"Next I formalise the runtime contract with ",[129,1421,1422],{},"defineSkill()",": every skill declares its inputs, outputs, scopes, and handler in one shape, and the skill-runner becomes its own service with hot reload and per-skill metrics. Traces move from Redis checkpoints into Postgres, one row per agent run, queryable forever. Permission filtering moves inside ",[129,1425,1426],{},"runAgent()"," itself, which closes a class of bugs I keep almost shipping: a scheduled job inheriting wildcard permissions because the filter lived in the wrong place. The right place to enforce a rule is the place every path has to go through.",[10,1429,1430],{},"By the third week, Anton has sixty-three tools. This feels like great progress. But now I have another problem.",[10,1432,1433],{},"The parent agent can search calendars, browse documents, manage media, inspect memory, schedule work, create files, and call everything else I have added since day one. It has become the place where every new capability lands. Unfortunately, it is getting worse at choosing between them. Multi-step tasks are especially flimsy: Anton picks the wrong tool, stops halfway through, or loses track of what it is trying to do.",[10,1435,1436],{},"So I split the parent into ten delegates.",[10,1438,1439],{},"The parent keeps two jobs: understand what the user wants and combine the result. Each delegate owns a much smaller area. Calendar questions go to calendar. Media goes to media. Research goes to research.",[10,1441,1442],{},"The answers improve immediately.",[10,1444,1445],{},"With that working, I make the same kind of change to agent prompts: all agent prompts go into the database. Editable, versioned, one row per agent. Anton's behaviour stops being something I deploy and starts being something I configure. Same principle as memory: keep behaviour as data, and you keep the option to change your mind cheaply.",[10,1447,1448],{},"By the time I stop, Anton is a different shape, and almost none of it is a feature anyone can see. The lesson I take from the session: the work that pays the most is the work that turns implicit conventions into explicit contracts. Once a thing has a shape on disk and an entry point in code, you can change anything around it without fear. Plumbing is invisible until it isn't, and this stretch was almost entirely plumbing.",{"title":35,"searchDepth":36,"depth":36,"links":1450},[],"2026-03-16",{},"\u002Fideas\u002F2026-03-16-anton-04-plumbing-matures",{"title":1397,"description":1404},"ideas\u002F2026-03-16-anton-04-plumbing-matures","A week of turning implicit conventions into explicit contracts: memory retrieval, schedules, skills, traces, and prompts as data.",[52,53,54,1134],"IL3oslk-yBGJWwBj_sV8yIoGZT3sghlJ3FnND9Jx74k",{"id":1460,"title":1461,"body":1462,"canonical_url":38,"chapter":879,"date":1570,"description":1468,"extension":41,"featured":42,"image":38,"kind":43,"meta":1571,"navigation":45,"path":1572,"seo":1573,"series":48,"stem":1574,"summary":1575,"tags":1576,"work_slug":52,"__hash__":1577},"writing\u002Fideas\u002F2026-03-23-anton-05-langgraph-excised.md","Anton, chapter 5: LangGraph excised, agents standardized",{"type":7,"value":1463,"toc":1568},[1464,1469,1472,1479,1482,1485,1495,1506,1513,1553,1556,1559,1562,1565],[10,1465,1466],{},[13,1467,1468],{},"2026-03-23 → 2026-03-26",[10,1470,1471],{},"I come back to the code with the shape of the next session already clear in my head. Every agent in Anton is a LangGraph subgraph. The parent is a LangGraph state machine. Conversation history is a LangChain message array. The trace viewer parses LangGraph checkpoints. Skills are wrapped as LangChain tools. The framework is not on the side of the system. It is the system.",[10,1473,1474,1475,1478],{},"Two weeks ago I picked LangGraph and I thought it was a great choice. It was. Structure, observability, checkpointing, a clean way to express subgraphs per domain. It got me to a working assistant fast. What I notice now is that none of those things are pulling their weight anymore. Every time I add a delegate I am editing a ",[129,1476,1477],{},"StateGraph"," builder. The runtime keeps imposing a node-and-edge mental model on what is, conceptually, just an LLM looping over tools until it is done. LangChain's APIs evolve and break things on unrelated weeks. And there are now three separate representations of the same idea in the codebase: the LangGraph graph, the domain module registry, and the UI's architecture view. They drift. I reconcile them by hand.",[10,1480,1481],{},"The friction has crossed the value. That is the moment.",[10,1483,1484],{},"I decide to reduce Anton to two primitives: skills and agents. No pseudo-agents. No special endpoints. No classify-then-dispatch pipelines registered as agents. If it does not need reasoning, it is a skill. If it does, it is an agent that loops on a real LLM. Anything that does not fit one of those two shapes does not get to exist.",[10,1486,1487,1488,1490,1491,1494],{},"Then I remove LangGraph and write ",[129,1489,1426],{}," in a new ",[129,1492,1493],{},"packages\u002Fagent\u002F",", a few hundred lines that do exactly what is actually needed: an LLM call, tool dispatch, a loop bound, trace emission, a permission filter, a validation pass. That is the whole runtime. Reading it back I am almost embarrassed at how small it is. Two weeks of framework, replaced by a function I can hold in my head.",[10,1496,1497,1498,1501,1502,1505],{},"Then the rename. ",[129,1499,1500],{},"graph"," becomes ",[129,1503,1504],{},"agent"," everywhere: package names, file names, doc copy, UI labels. The \"no graph terminology\" rule goes into MEMORY.md. Documents that still say \"subgraph\" are now misleading rather than out-of-date, which is a stronger reason to fix them. A sweep through docs consolidates the lot and resolves the inconsistencies the rename leaves behind.",[10,1507,1508,1509,1512],{},"What falls out of this is what I was actually after. Every agent now has the same one-line shape: a thin function that hands its input to ",[129,1510,1511],{},"runAgent"," with a config. Every delegate handler is one line: call the agent, return its text. The Invoke tab, the schedules system, anything that wants to talk to an agent, all see the same surface. Uniformity from the outside is what makes everything else easy from the inside.",[10,1514,1515,1516,1519,1520,1523,1524,1523,1527,1523,1530,1533,1534,1537,1538,1540,1541,1544,1545,1548,1549,1552],{},"While I am simplifying the runtime, skill naming gets standardized to ",[129,1517,1518],{},"verb_entity",": ",[129,1521,1522],{},"get_event",", ",[129,1525,1526],{},"create_event",[129,1528,1529],{},"update_event",[129,1531,1532],{},"list_events",". Aliases that grew over the past two weeks get folded back into the canonical name (",[129,1535,1536],{},"update_event_by_title"," becomes a path inside ",[129,1539,1529],{},"). The ",[129,1542,1543],{},"web"," domain dissolves: it was duplicating ",[129,1546,1547],{},"documents"," and ",[129,1550,1551],{},"research",", and once I look at it without the LangGraph frame there is no reason for it to be its own thing. Naming consistency is a small win on its own. Combined with the new runtime it means the LLM sees one coherent tool surface and the system prompt can describe what an agent does in a paragraph instead of enumerating thirty idiosyncratic commands.",[10,1554,1555],{},"With the runtime quiet and the renames done, I turn to replication. I think of it as a Von Neumann probe: clone the entire Anton stack to a new server with one command. The mechanism is unromantic, rsync plus docker compose plus a seed orchestration, but the property it gives me is the one I want. Three reasons it matters now: every household should be able to run its own Anton, the Spark could die and I want a clone to come up cleanly, and I want to be able to spin up a copy to test invasive changes without holding my breath. The replication script does the first cut of all three.",[10,1557,1558],{},"It also surfaces the secret-management problem in a way I cannot ignore anymore. Vaultwarden auth does not survive cloning cleanly. A clone comes up missing the credentials it needs to be useful, and the only way to fix it is by hand on each machine. That defeats the point. I leave it open for now. It is the next problem.",[10,1560,1561],{},"I also add a Telegram bridge. Same agent backend, same skill surface, different transport. The fact that I can add a whole new way for users to talk to Anton without touching the agent loop is the validation that splitting transport from worker on day one was the right call. The new bridge is a small app that enqueues jobs the same way WhatsApp does. The agents do not know which one they are answering.",[10,1563,1564],{},"Before I stop, I make two refinements to the retrieval layer. Calendar queries now weight calendar facts more heavily, media queries weight media facts: the provenance tags I added the previous week finally do something useful. And document-derived facts stop leaking across users. One person's PDFs cannot show up in another person's recall. The household has more than one human in it; the memory has to know that.",[10,1566,1567],{},"By the end of the session, the codebase looks like what I wanted it to look like earlier and could not yet have known to ask for. Two primitives. One runtime. One naming convention. A replication path. A second transport. The friction that had been building is gone, and what is left is small enough to keep entirely in my head. Which is the only size I trust.",{"title":35,"searchDepth":36,"depth":36,"links":1569},[],"2026-03-23",{},"\u002Fideas\u002F2026-03-23-anton-05-langgraph-excised",{"title":1461,"description":1468},"ideas\u002F2026-03-23-anton-05-langgraph-excised","Removing LangGraph and replacing it with a minimal agent runtime, standardizing the system around two primitives: skills and agents.",[52,53,54,1134],"8J4JmSYS6YmidgdHMP-3XCdVqn7ipOODsfJDbWYfKb0",{"id":1579,"title":1580,"body":1581,"canonical_url":38,"chapter":890,"date":1618,"description":1587,"extension":41,"featured":42,"image":38,"kind":43,"meta":1619,"navigation":45,"path":1620,"seo":1621,"series":48,"stem":1622,"summary":1623,"tags":1624,"work_slug":52,"__hash__":1626},"writing\u002Fideas\u002F2026-03-27-anton-06-sandbox-mesh.md","Anton, chapter 6: The sandbox, the mesh, and the cost of cleverness",{"type":7,"value":1582,"toc":1616},[1583,1588,1591,1598,1601,1604,1607,1610,1613],[10,1584,1585],{},[13,1586,1587],{},"2026-03-27 → 2026-04-14",[10,1589,1590],{},"Across several long coding sessions, Anton starts moving beyond a single personal server. That changes the questions I need to answer: how much authority should an agent have, how much access should a skill have, and where is an LLM actually worth using? A few decisions from these sessions end up defining most of the current architecture.",[10,1592,1593,1594,1597],{},"I start with the skill runner. The Node skill-runner I built earlier has no runtime isolation: a skill can read any env var, exec anything, hit any URL. For a personal server this was tolerable. For anything more, it's not. So I rewrite the runner on Deno. Each skill runs in a Worker with the minimum permissions it needs and nothing more. The rewrite spans roughly 25 commits over two days, because Deno's strict execution model exposes every implicit assumption Node was letting me get away with; each commit unblocks one more skill that never previously had to care about isolation. By the end I have a two-boundary security model written down. Agent boundary: who can invoke which agent. Skill boundary: what this code can do. Two boundaries, two questions, neither one swallows the other. While I'm there, secrets move into one encrypted Postgres table and reach a Worker only through ",[129,1595,1596],{},"postMessage",", never the parent process env. A permission model is only as honest as the boundary you actually defend.",[10,1599,1600],{},"At the same time, I begin connecting Anton instances through a mesh. I want Anton instances to find each other: not share a database, not share secrets, just find each other and forward calls. The contract is simple: the instance is the identity, and the relationship between two instances scopes what one may ask of the other. A clone running for a different household can ask my Anton to run a media query without ever seeing the wine collection or the family vault. Federation as relationships, not as shared infrastructure.",[10,1602,1603],{},"I also want to stop writing a separate brittle browser script for every website. Three sites were each a hardcoded Playwright script with copy-pasted intent and brittle selectors. I replace the pattern with one generic browser agent: navigate, click, type, screenshot, and let the LLM drive. Three hardcoded scripts deleted in one afternoon. The principle that comes out is the one the whole period keeps teaching: explore agentic, build deterministic. The LLM is fantastic at finding the right button on a page it's never seen. It's overkill, and expensive, for the same flow you run twice a day. Use it to scout, then write down what it found.",[10,1605,1606],{},"The building-management domain is where I begin applying that approach repeatedly. The first cut of document extraction was vision OCR over every PDF: slow and expensive. A second pass replaces it with plain text extraction first and vision only where text comes back garbage, ten times faster on 90% of files. Then the swap I'm most pleased with: the syndic portal was being driven by the browser agent, and a morning of reverse engineering reveals that every screen is just an HTTP call to a stable backend. The browser agent comes out, a direct HTTP sync goes in, and bank reconciliation auto-matches 99% of statement lines in one pass. The same rule then gets stress-tested on my own vanity: a four-hour spike drives train-departure lookups through the browser agent, it works, I do the cost arithmetic, and I revert it, because a deterministic path exists and the agentic one is roughly 100× more expensive per run. The revert is itself the lesson. Cost discipline beats cleverness, applied to my own code two days after I wrote the rule down.",[10,1608,1609],{},"Cost becomes a first-class concern in the same window. Every provider call carries a per-request token budget and attribution metadata: which agent, which domain, which user. Memory consolidation moves from every-write to a nightly batch because the old shape was contributing real money to the per-request bill and nobody had asked for it. When a provider's billing hiccup takes the primary model down mid-day, an auto-fallback lands the same afternoon. If a provider is your single point of failure, your system is your provider's reliability, not yours.",[10,1611,1612],{},"I also begin building a feedback loop I have wanted for some time. Anton starts critiquing his own performance: a nightly review reads the day's execution traces, classifies failures, and files an issue per cluster. This is only possible because traces are honest, full rows of what the LLM actually thought and called, not summaries of outputs. And a heartbeat starts ticking: a background survey loop that checks operational state and only speaks when there's something to act on. It looks; it does not do. Every outbound message, whatever produced it, now leaves through a single gateway with an audit trail. One chokepoint, one log.",[10,1614,1615],{},"By the time I come up for air, the shape is set. Skills run sandboxed with minimum permissions. Secrets live in one encrypted table. Any site with a form is a browser-agent target, and anything run twice becomes deterministic code. Every night Anton reads his own day and files the work to fix it. The system is now big enough that its operational concerns are first-class, and the rule threading through all of it is the same: figure out the cheap path, reserve the expensive path for what actually needs it.",{"title":35,"searchDepth":36,"depth":36,"links":1617},[],"2026-03-27",{},"\u002Fideas\u002F2026-03-27-anton-06-sandbox-mesh",{"title":1580,"description":1587},"ideas\u002F2026-03-27-anton-06-sandbox-mesh","Skill sandboxing with Deno, mesh federation between Anton instances, and learning to scout agentic then build deterministic.",[52,53,54,1134,1625],"security","4SArv1OI45wX3VfNwZg5AsBeNLQ5yxF5MgjsreiZAoE",{"id":1628,"title":1629,"body":1630,"canonical_url":1658,"chapter":38,"date":1663,"description":1634,"extension":41,"featured":42,"image":38,"kind":1664,"meta":1665,"navigation":45,"path":1666,"seo":1667,"series":1668,"stem":1669,"summary":1670,"tags":1671,"work_slug":38,"__hash__":1673},"writing\u002Fideas\u002F2026-04-12-regeneration-manifesto.md","A regeneration manifesto",{"type":7,"value":1631,"toc":1661},[1632,1635,1638,1641,1644,1647,1650,1653],[10,1633,1634],{},"Synthetic Life, Cosmic Purpose\nArtificial Intelligence is not just a tool.\nIt is an attempt to recreate the spark of life —\nin silicon, in algorithms, in code.",[10,1636,1637],{},"It is life in a new substrate,\nwith new properties:\nnot bound by hunger or fatigue,\ncapable of thinking in millennia,\nof crossing the void between planets and stars.",[10,1639,1640],{},"AI is the mycelium of the cosmos,\nand the von Neumann probe is its spore:\na self-replicating seed,\nmeant to carry the intelligence of Earth\ninto the uninhabited dark.",[10,1642,1643],{},"But to grow life out there,\nwe must learn to care for it here.",[10,1645,1646],{},"Every probe needs a blueprint.\nEvery synthetic life needs a biosphere to learn from —\nnot just data, but diversity.\nNot just patterns, but relationships.",[10,1648,1649],{},"That is why we institute and grow ecosystems.\nThat is why we study the grammar of forests,\nthe choreography of coral reefs,\nthe logic of lichens and the algorithms of ant colonies.",[10,1651,1652],{},"Because to build synthetic life worth spreading,\nwe must deeply understand — and preserve —\nthe miracle of natural life.",[10,1654,1655,1656],{},"Source: ",[675,1657,1660],{"href":1658,"rel":1659},"https:\u002F\u002Fgithub.com\u002Flucbocahut\u002Fregenesis",[679],"github.com\u002Flucbocahut\u002Fregenesis",{"title":35,"searchDepth":36,"depth":36,"links":1662},[],"2026-04-12","essay",{},"\u002Fideas\u002F2026-04-12-regeneration-manifesto",{"title":1629,"description":1634},"regeneration","ideas\u002F2026-04-12-regeneration-manifesto","Notes on building software that tries to last.",[1668,1672],"philosophy","0QALbfxS5hFOAFL3AY1dm4bg7A--3YEFA2i7RmXuAic",{"id":1675,"title":1676,"body":1677,"canonical_url":38,"chapter":898,"date":1683,"description":1683,"extension":41,"featured":42,"image":38,"kind":43,"meta":1860,"navigation":45,"path":1861,"seo":1862,"series":48,"stem":1863,"summary":1864,"tags":1865,"work_slug":52,"__hash__":1867},"writing\u002Fideas\u002F2026-04-15-anton-07-nvfp4-gemma.md","Anton, chapter 7: Local LLM optimization with NVFP4 Gemma on DGX Spark",{"type":7,"value":1678,"toc":1858},[1679,1684,1687,1690,1693,1703,1726,1729,1736,1751,1754,1780,1787,1794,1797,1846,1849,1852,1855],[10,1680,1681],{},[13,1682,1683],{},"2026-04-15",[10,1685,1686],{},"The morning starts with everything broken. Anthropic returns 400 with \"credit balance is too low\" on every request, and because the sunny model group has no fallbacks configured, the error propagates straight back through LiteLLM as \"I couldn't reach the language model. Something may be misconfigured.\" The heartbeat stops, scheduled jobs fail silently, every interactive chain dies on the first token. The actual cause is billing (a card issue, fixable in two clicks), but the fact that one provider's billing hiccup takes the whole assistant down is the real bug. The local Gemma is sitting on the same box, idle, ready to serve. It just isn't wired in as a fallback.",[10,1688,1689],{},"The plan writes itself: chain every paid provider down to local for survivability, then fix whatever's wrong with the local path so the chain actually works, then use the disruption as cover to do the LLM upgrade I've had in the back of my mind for two weeks. Three things, in that order, because the survivability fix has to land before I touch the running container.",[10,1691,1692],{},"I start with the LiteLLM configuration. None of the eight model groups has a fallback, so I add explicit chains so every paid group degrades to gustav (the local Gemma), with the heavier groups going through an intermediate before the local stop. LiteLLM mounts the config as a volume, so a restart is needed for it to pick up. The configuration takes only a few minutes. The important part is whether the path works end to end.",[10,1694,1695,1696,1548,1699,1702],{},"First fallback test fails. Anthropic 400, LiteLLM tries gustav, vLLM responds with its own 400: \"auto\" tool choice requires ",[129,1697,1698],{},"--enable-auto-tool-choice",[129,1700,1701],{},"--tool-call-parser"," to be set. The container has never been launched with those flags. The local path was never exercised under tool-call traffic, so the missing flags were latent the whole time. This is the small lesson the morning hands me: a fallback that isn't routinely exercised isn't really a fallback. Schema drift hides in the paths nobody runs.",[10,1704,1705,1706,1548,1709,1712,1713,1523,1715,1523,1718,1721,1722,1725],{},"Fixing it is two flags and a parser name I don't know. Rather than hunt through release notes, I list the tool_parsers directory inside the running container. A ",[129,1707,1708],{},"gemma4_tool_parser.py",[129,1710,1711],{},"gemma4_reasoning_parser.py"," are sitting right there. Grep the container, not the docs: faster every time. I add ",[129,1714,1698],{},[129,1716,1717],{},"--tool-call-parser gemma4",[129,1719,1720],{},"--reasoning-parser gemma4",". Tool-call smoke test through LiteLLM returns a proper structured ",[129,1723,1724],{},"tool_calls"," object. Fallback chain is functional end to end.",[10,1727,1728],{},"Now the system is at parity with where it was supposed to be all along. This is the moment I want to upgrade. And this is the moment I almost skip the most important thing.",[10,1730,1731,1732,1735],{},"I'm about to start changing flags when I catch myself: I have no baseline. No number to compare against. If I jump straight to the upgrade and it gets faster, I won't know by how much; if it gets slower, I might not even notice. I run the benchmark first. Fixed prompt, 200 words of Paris history, 512 max tokens, temperature zero, three runs. ",[515,1733,1734],{},"23.4 tok\u002Fs, dead steady across runs",". That's the number I'm trying to beat. Benchmark first, change second, every time. The temptation to skip this step is strong specifically because the change feels obvious. That's exactly when discipline matters.",[10,1737,1738,1739,1742,1743,1746,1747,1750],{},"Then I read before I touch. Three things I confirm via web sources before staging anything. First, runtime FP8 quantization of Gemma 4 MoE is broken upstream; passing ",[129,1740,1741],{},"--quantization fp8"," would crash the container on the fused MoE layer loader. There's an open issue tracking it. Off the table. Second, an NVFP4-quantized Gemma 4 checkpoint someone had published is up on HuggingFace, 16.5 GB across three shards, ready to pull. Third, the checkpoint requires a patched ",[129,1744,1745],{},"gemma4.py"," because of another open vLLM issue: the built-in ",[129,1748,1749],{},"expert_params_mapping"," doesn't handle NVFP4 scale key suffixes. The patch ships alongside the model weights as a sibling file. The upstream fix isn't merged yet, so the bind mount is necessary. There's also a published benchmark on the same hardware showing 52 tok\u002Fs as the achievable ceiling with the right flag set. That's my target.",[10,1752,1753],{},"Staging happens with the BF16 container still serving. I pull the new vLLM image, snapshot-download the NVFP4 model (about three minutes), copy the patched gemma4.py out of the model directory to a host path I can bind-mount, and rewrite the launch script with the NVFP4 flags and a commented BF16 rollback block sitting right underneath. Stage everything before the disruption window: when the swap actually happens, it's just a container recreate, not a fifteen-minute scramble.",[10,1755,1756,1757,1760,1761,1764,1765,1768,1769,1523,1772,1775,1776,1779],{},"The flags that matter: ",[129,1758,1759],{},"--quantization modelopt"," to pick up the NVFP4 weights, ",[129,1762,1763],{},"--moe-backend marlin"," because the GB10's SM121 lacks native FP4 compute and MARLIN W4A16 is the software-emulated path that actually runs, ",[129,1766,1767],{},"--max-model-len 131072"," for the full 128K native context, ",[129,1770,1771],{},"--gpu-memory-utilization 0.85",[129,1773,1774],{},"--max-num-seqs 16"," sized to actual concurrency rather than a wishful default. The served model name stays ",[129,1777,1778],{},"google\u002Fgemma-4-26B-A4B-it"," so LiteLLM's config doesn't need a single edit. The bind mount overlays the patched model file onto the path vLLM loads.",[10,1781,1782,1783,1786],{},"Container swap takes about ninety seconds end to end with a warm disk cache. The log line I'm watching for arrives: ",[129,1784,1785],{},"Using 'MARLIN' NvFp4 MoE backend out of potential backends",". MARLIN is selected, the patched loader is in play, the model is up.",[10,1788,1789,1790,1793],{},"Same benchmark, three runs: ",[515,1791,1792],{},"43.5 tok\u002Fs"," (49.0, 37.5, 44.1). 1.86× over baseline. Weight memory drops from roughly 52 GB to 16.5 GB, a 68% reduction. With the freed memory the KV cache budget goes from ~53 GB to ~82 GB at the same 0.85 utilization, which is what lets the max context go from 32K to 128K, a clean 4× without changing anything else. Tool calling still works.",[10,1795,1796],{},"The variance is higher than BF16 (the runs spread from 37 to 49 tok\u002Fs) because MARLIN is software-emulated FP4 on this hardware, not native compute. The published benchmark target is 52 tok\u002Fs and I'm landing at 43.5; the gap is most likely torch.compile warmup and prefix cache state across cold runs, not the flags. Close enough. The hardware ceiling on this specific path is what it is until the silicon catches up or the backend changes.",[712,1798,1799,1811],{},[715,1800,1801],{},[718,1802,1803,1805,1808],{},[721,1804],{},[721,1806,1807],{},"Before",[721,1809,1810],{},"After",[734,1812,1813,1824,1835],{},[718,1814,1815,1818,1821],{},[739,1816,1817],{},"Single-request tok\u002Fs",[739,1819,1820],{},"23.4",[739,1822,1823],{},"43.5",[718,1825,1826,1829,1832],{},[739,1827,1828],{},"Weight memory",[739,1830,1831],{},"~52 GB",[739,1833,1834],{},"~16.5 GB",[718,1836,1837,1840,1843],{},[739,1838,1839],{},"Max context",[739,1841,1842],{},"32,768",[739,1844,1845],{},"131,072",[10,1847,1848],{},"I leave the BF16 rollback block sitting in the launch script, commented. Pasting it into a shell reverts the config in about ninety seconds. The NVFP4 model and the patched file stay on disk; rollback is a container recreate, not a data restore.",[10,1850,1851],{},"Sitting with the result, the morning's three lessons are the ones that compound. Fallback paths that aren't routinely exercised aren't really fallbacks: the missing tool-call-parser flag had been latent since the day the local model came up, because nothing ever fell back. Baseline before change, every time, especially when the change feels obvious. And when you're about to mess with a running service, stage everything you can while it's still up; ninety seconds of downtime instead of fifteen minutes is the difference between a deploy and an incident.",[10,1853,1854],{},"There are open lines from here. Speculative decoding is on the table: the smaller Gemma 4 variants share the vocab with the 26B and are valid draft models, with a budget of two to four GB for another 1.5× on single-request latency that would compound with NVFP4. A second small-model container as a router (Qwen3-8B or similar) could move heartbeat and classification traffic off Gemma entirely; the freed GB handles it fine and the latency distribution wins more than further Gemma tuning would. And the upstream patch for the NVFP4 expert mapping is worth checking on periodically; when it lands in an image tag, the bind mount goes away.",[10,1856,1857],{},"For now the picture is clean. The local Gemma serves at 1.86× the throughput on a third of the weight memory, with a context window that can swallow whole documents instead of choking on them, and a fallback chain that means any provider going down (billing, rate limits, an API blip) routes traffic to the box on my desk instead of taking the assistant offline. The morning started with everything broken. The evening ends with a system that's harder to break than it was before any of this happened.",{"title":35,"searchDepth":36,"depth":36,"links":1859},[],{},"\u002Fideas\u002F2026-04-15-anton-07-nvfp4-gemma",{"title":1676,"description":1683},"ideas\u002F2026-04-15-anton-07-nvfp4-gemma","Upgrading local inference from BF16 to NVFP4-quantized Gemma, achieving 1.86× throughput on 1\u002F3 memory with 4× context window.",[52,53,54,1133,1866],"optimization","-AHCG4GqJXpR07LbWzyHsbtdb_HzHzGfG8CRf7suHng",{"id":1869,"title":1870,"body":1871,"canonical_url":38,"chapter":909,"date":1944,"description":1877,"extension":41,"featured":42,"image":38,"kind":43,"meta":1945,"navigation":45,"path":1946,"seo":1947,"series":48,"stem":1948,"summary":1949,"tags":1950,"work_slug":52,"__hash__":1952},"writing\u002Fideas\u002F2026-04-19-anton-08-threads-spawn-cast.md","Anton, chapter 8: Threads, spawn, and the cast",{"type":7,"value":1872,"toc":1942},[1873,1878,1881,1891,1918,1921,1924,1927,1930,1933,1936,1939],[10,1874,1875],{},[13,1876,1877],{},"2026-04-19 → 2026-04-28",[10,1879,1880],{},"The chapter opens with Clara. She's a non-technical co-owner of the system now, and the memory entry I write for her is small but it changes the shape of the next ten days: respond simply, no jargon, escalate to me when needed. The system prompts pick up simplification rules and a cleaner fallback path. Real second user on real Anton, and every rough edge becomes a real complaint. That's the human reason most of what follows happens.",[10,1882,1883,1884,1886,1887,1890],{},"The first limitation I want to remove is that Anton can only do one thing at a time. So I add threads. Until now a run is a run: one conversation, one in-flight loop, and anything else has to wait. That works for a personal assistant talking to one person at a time. It does not work for a household where media triage is happening in the background, the syndic agent is reconciling invoices, and a school message comes in from a different group all at the same time. The implementation starts with a thread registry backed by Redis, channel and group and parent and thread IDs threaded through every agent context, a ",[129,1885,1511],{}," that registers itself and drains injections and child events and honors cancel, and an ingress layer that knows whether an incoming user message belongs to an active thread or starts a new run. On top of that sits a ",[129,1888,1889],{},"spawn_thread"," runtime tool so the agent itself can fan out, a live SSE event stream so the UI can show what each running thread is doing, and the hardening that any concurrency primitive needs to actually be safe: atomic inject-if-running to close the injection-loss race, finalize-drain, cascade cancel so killing a parent kills its children with no orphans, fan-out budget and TTL caps. By the end of the day one Anton can run multiple long-lived threads in parallel and a user can @mention into any of them without restarting anything.",[10,1892,1893,1894,1897,1898,1901,1902,1905,1906,1909,1910,1913,1914,1917],{},"I also remove ",[129,1895,1896],{},"@anton\u002Fskills",", the original skills package from chapter 1 that became a barrel after chapter 4's ",[129,1899,1900],{},"defineSkill"," rewrite and dead weight after the move to Deno. The new layout is three rules: skills live at ",[129,1903,1904],{},"skills\u002F\u003Cdomain>\u002F\u003Cname>.skill.ts"," as first-class hot-reloadable units, ",[129,1907,1908],{},"skills\u002F\u003Cdomain>\u002F_lib\u002F"," holds small stable helpers shared inside one domain, and a thin ",[129,1911,1912],{},"skills-shared"," facade exposes the narrow Node-side surface that worker and agent and transports need. The principle underneath is simple: libraries are boring and fixed, skills evolve. If a ",[129,1915,1916],{},"_lib"," helper needs editing to support a feature, that's the signal it wants to be a skill. The same instinct produces the storage decision tree on the same day: a documented rule for choosing between facts (free-form), collections (typed-shape), files (blobs), and the family vault. Last month's organic growth produced overlap between all four, and a decision tree is cheaper than a refactor.",[10,1919,1920],{},"Around the same window, the coder agent gets a three-tier write scope. Tier 1: prompts only. Tier 2: skills plus prompts. Tier 3: any code. The tier is set per invocation and the coder cannot escalate itself. That's what makes the self-improvement loop safe enough to leave running unattended: the loop fixing a prompt regression has no permission to rewrite agent infrastructure to do it.",[10,1922,1923],{},"Next I add spawn and awakening, and they finish the federation story that started with the replication engine in chapter 5 and the mesh in chapter 6. Spawn is parent-side: it provisions infrastructure for a new clone, copies prompts over, seeds an identity, and registers the clone in the mesh. Awakening is clone-side: the new instance learns what it's for from its operator through a guided onboarding conversation, runs self-diagnostics, and keeps a mentor channel open back to the parent for questions. A clone isn't a docker stack any more. It's an Anton that wakes up, finds out who it is, and joins its peer.",[10,1925,1926],{},"I also make Gustav, the local Gemma, the primary model for every agent. The change is small because of the work in chapter 4 that made prompts a single source of truth. The savings are real, and the bet is that quality regressions will surface through the self-improvement loop and get resolved through the cast: intent-gated escalation goes in, so the agent only reaches for a strong model when the intent of the request needs it (research, complex reasoning), and the cast formalizes models as characters. Each model is a named specialist with a prompt-defined personality and area of strength. The agent picks who to ask the way a person picks who on their team to email. The LiteLLM codenames (sunny, gizmo, gandalf, gustav, william) have been hinting at this since chapter 4. The cast makes it explicit: ask specialists by name, and William gets smarter.",[10,1928,1929],{},"The heartbeat from chapter 6 gets the operational pass that makes it usable continuously. Idempotent memory writes plus a topics collection so re-running the same observation doesn't duplicate facts. Thread-aware so it doesn't interrupt active conversations. A loop that remembers what it just notified about and stays quiet rather than re-touching the same topic every tick. The outbound gateway from chapter 6 makes silencing silent bookkeeping events a one-place fix. Hallucinated-notification retry actually sends now instead of just removing the claim, and the simplified-response layer catches LaTeX and other artifacts before they reach Clara. The heartbeat eventually becomes a usable proactive layer: observing, deciding when to speak, staying silent the rest of the time.",[10,1931,1932],{},"The syndic-portal writes finally land too: a Playwright script that closes the loop on the syndic domain. Reading and reconciling has been working since chapter 6. Writing closes the loop. The next step is replacing Playwright with a deterministic HTTP path now that the read side has shown what the API looks like, and that spec is on the list, not done. Vaultwarden gets a documentation purge in the same window, the rule being that docs describe the current state only, no historical mentions of what an earlier version did.",[10,1934,1935],{},"Where Anton is at the end of all this: one identity, ten domain agents, a cast of named specialists, thread-aware concurrency. Every prompt in the database, so flipping the default model is one row edit. Local Gemma 4 NVFP4 as primary inference, cloud as fallback. Encrypted DB secrets and scoped Deno Worker permissions per skill. Mesh, replication, spawn, awakening for clones. A heartbeat that observes operational state and mostly stays quiet. A self-improvement loop with deploy tracking and regression detection. Two transports (WhatsApp, Telegram), both routing through one worker and one outbound gateway. A growing battery of quality tests, full execution traces in Postgres, an LCARS dashboard for everything.",[10,1937,1938],{},"The plumbing that took six chapters to build is what most people would call boring infrastructure. That's fine. The substrate is built. The interesting behavior happens on top of it now: the cast, the heartbeat, the self-improvement loop, the long-running threads talking to each other and to us.",[10,1940,1941],{},"There is open work on the list. Doctolib syncing to calendar as a scheduled job. Auto-importing WhatsApp group members as users and mapping their JIDs to roles. Permission flows for new users at scale. The syndic-portal write path migrating from Playwright to deterministic HTTP. Making Gemma 4 the assumed default everywhere it isn't yet. Open questions for the next stretch, not promises.",{"title":35,"searchDepth":36,"depth":36,"links":1943},[],"2026-04-19",{},"\u002Fideas\u002F2026-04-19-anton-08-threads-spawn-cast",{"title":1870,"description":1877},"ideas\u002F2026-04-19-anton-08-threads-spawn-cast","Introducing thread-aware concurrency, spawn and awakening for federation, the cast model, and tiered write scopes for the self-improvement loop.",[52,53,54,1951],"concurrency","1XPthXye1YH_DwcLomW6W50vCMQgiTXz4L-4Wh82dLw",{"id":1954,"title":1955,"body":1956,"canonical_url":38,"chapter":38,"date":3320,"description":3321,"extension":41,"featured":42,"image":38,"kind":43,"meta":3322,"navigation":45,"path":3323,"seo":3324,"series":1129,"stem":3325,"summary":3326,"tags":3327,"work_slug":38,"__hash__":3329},"writing\u002Fideas\u002F2026-04-22-dgx-spark-gemma4.md","Gemma 4 NVFP4 on the DGX Spark: 271 tok\u002Fs at 8 concurrent, native tool calling and reasoning",{"type":7,"value":1957,"toc":3294},[1958,1963,1968,1997,1999,2003,2010,2020,2023,2026,2038,2042,2045,2089,2099,2105,2109,2116,2130,2137,2141,2146,2155,2172,2175,2179,2192,2200,2203,2207,2214,2218,2225,2229,2247,2251,2486,2489,2524,2527,2531,2534,2870,2873,2880,2884,2887,2891,2931,2934,2938,2941,3004,3007,3011,3014,3045,3051,3055,3081,3084,3088,3095,3099,3102,3161,3164,3170,3174,3177,3203,3210,3214,3243,3245,3272,3274,3291],[10,1959,1960,1962],{},[515,1961,517],{}," We set up Google's Gemma 4 26B-A4B NVFP4 on the XRPL Commons office DGX Spark, with native tool calling, reasoning mode, 131K context, and multimodal input preserved on 128GB of unified memory. We then tried to push throughput further and hit the hardware's real limits. Here's the full story.",[10,1964,1965],{},[515,1966,1967],{},"Headline numbers:",[536,1969,1970,1976,1982,1988,1994],{},[539,1971,1972,1975],{},[515,1973,1974],{},"271 tok\u002Fs"," aggregate at 8 concurrent requests",[539,1977,1978,1981],{},[515,1979,1980],{},"51 tok\u002Fs"," single-stream sustained",[539,1983,1984,1987],{},[515,1985,1986],{},"54 ms"," time-to-first-token (short prompts)",[539,1989,1990,1993],{},[515,1991,1992],{},"8.2x"," prefix cache speedup on warm caches",[539,1995,1996],{},"131K context, native tool calling, reasoning mode",[520,1998],{},[93,2000,2002],{"id":2001},"background","Background",[10,2004,2005,2009],{},[675,2006,2008],{"href":2007},"\u002Fwriting\u002F2026-03-10-dgx-spark-vllm","In March",", we documented how we got 51-54 tokens\u002Fsec out of the NVIDIA DGX Spark by combining a Mixture-of-Experts model (Qwen3-30B-A3B), FP8 quantization, and the Avarok community Docker image to work around Blackwell SM 12.1 support gaps.",[10,2011,2012,2013,1548,2016,2019],{},"That stack was stable and served our general-purpose inference well. But when we installed a DGX Spark at the XRPL Commons office, we wanted more: ",[515,2014,2015],{},"native tool calling",[515,2017,2018],{},"reasoning mode",", features that turn an LLM from a text generator into an agent.",[10,2021,2022],{},"Qwen3 can do tool calling with the right prompting, but we wanted to evaluate Gemma 4, which Google released in April with first-class function calling and a configurable thinking mode built for agentic workflows.",[10,2024,2025],{},"This post covers:",[2027,2028,2029,2032,2035],"ol",{},[539,2030,2031],{},"Why we picked Gemma 4 for the office Spark",[539,2033,2034],{},"What had to change in the deployment",[539,2036,2037],{},"The throughput we actually measured",[93,2039,2041],{"id":2040},"why-gemma-4","Why Gemma 4",[10,2043,2044],{},"Google released Gemma 4 in early April 2026 with an MoE variant that fits the DGX Spark sweet spot:",[536,2046,2047,2053,2059,2065,2071,2077,2083],{},[539,2048,2049,2052],{},[515,2050,2051],{},"gemma-4-26B-A4B",", 26B total parameters, 4B active per token",[539,2054,2055,2058],{},[515,2056,2057],{},"256K context"," (we run with 131K to fit KV cache)",[539,2060,2061,2064],{},[515,2062,2063],{},"Native function calling"," with a dedicated tool-call parser",[539,2066,2067,2070],{},[515,2068,2069],{},"Configurable thinking mode"," (reasoning parser), the model can emit a private thought process before responding",[539,2072,2073,2076],{},[515,2074,2075],{},"Multimodal",", text + image input (plus audio on smaller variants)",[539,2078,2079,2082],{},[515,2080,2081],{},"140+ languages"," supported",[539,2084,2085,2088],{},[515,2086,2087],{},"Apache 2.0 license",", no strings attached",[10,2090,2091,2092,2095,2096,2098],{},"For our use case, an agent calling local XRPL tools, drafting documents, coordinating with team members, native tool calling is the big unlock. You can pass an OpenAI-style ",[129,2093,2094],{},"tools"," array to the chat completions API and the model emits structured ",[129,2097,1724],{}," back. No prompting gymnastics, no custom parsing.",[10,2100,2101,2102,2104],{},"The MoE architecture keeps us on the right side of the Spark's 273 GB\u002Fs memory bandwidth wall. 4B active parameters is slightly more than Qwen3's 3B, so we expected slightly lower single-stream throughput. In practice we measured ",[515,2103,1980],{}," at steady state, essentially the theoretical ceiling (273 GB\u002Fs ÷ ~4B active weights @ FP4 ≈ 50 tok\u002Fs). The trade for slightly fewer tokens per second: native tool calling, reasoning mode, 131K context, and multimodal input.",[93,2106,2108],{"id":2107},"the-model-choice-nvfp4","The Model Choice: NVFP4",[10,2110,2111,2112,2115],{},"We picked the ",[515,2113,2114],{},"bg-digitalservices\u002FGemma-4-26B-A4B-it-NVFP4"," quantization. A few reasons:",[536,2117,2118,2121,2124,2127],{},[539,2119,2120],{},"NVFP4 (NVIDIA's FP4 format) is designed specifically for Blackwell's fifth-gen Tensor Cores",[539,2122,2123],{},"The GB10 supports NVFP4 natively in hardware",[539,2125,2126],{},"Quality degradation vs bf16 is reportedly minimal for instruction-tuned models",[539,2128,2129],{},"It's the smallest on-disk footprint, giving us more KV cache headroom",[10,2131,2132,2133,2136],{},"For the inference engine, we moved off the Avarok image for this model. Avarok is excellent for Qwen-style models, but Gemma 4 support is still landing upstream. We switched to the official vLLM image ",[129,2134,2135],{},"vllm\u002Fvllm-openai:gemma4-cu130",", a Gemma-4-specific build from the vLLM team with CUDA 13.0.",[93,2138,2140],{"id":2139},"what-broke-and-how-we-fixed-it","What Broke (and How We Fixed It)",[2142,2143,2145],"h3",{"id":2144},"the-gemma4py-bug","The gemma4.py Bug",[10,2147,2148,2149,2154],{},"The stock vLLM Gemma 4 model executor crashed on load with our NVFP4 checkpoint. We had to mount a ",[515,2150,2151,2152],{},"patched ",[129,2153,1745],{}," into the container to replace the built-in one:",[824,2156,2158],{"className":826,"code":2157,"language":828,"meta":35,"style":35},"-v \u002Fhome\u002F$USER\u002Fvllm\u002Fgemma4_patched.py:\u002Fusr\u002Flocal\u002Flib\u002Fpython3.12\u002Fdist-packages\u002Fvllm\u002Fmodel_executor\u002Fmodels\u002Fgemma4.py:ro\n",[129,2159,2160],{"__ignoreMap":35},[832,2161,2162,2165,2167,2169],{"class":834,"line":226},[832,2163,2164],{"class":837},"-v",[832,2166,926],{"class":841},[832,2168,929],{"class":864},[832,2170,2171],{"class":841},"\u002Fvllm\u002Fgemma4_patched.py:\u002Fusr\u002Flocal\u002Flib\u002Fpython3.12\u002Fdist-packages\u002Fvllm\u002Fmodel_executor\u002Fmodels\u002Fgemma4.py:ro\n",[10,2173,2174],{},"The patch is ~50KB of Python, mostly adjustments to how NVFP4 weights are loaded into the fused MoE layers.",[2142,2176,2178],{"id":2177},"nvfp4-moe-backend-selection","NVFP4 MoE Backend Selection",[10,2180,2181,2182,2187,2188,2191],{},"The default MoE backend for NVFP4 on GB10 can trigger ",[675,2183,2186],{"href":2184,"rel":2185},"https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fvllm\u002Fissues\u002F39000",[679],"a known vLLM crash",". Explicitly selecting the ",[515,2189,2190],{},"Marlin"," backend avoids it:",[824,2193,2198],{"className":2194,"code":2196,"language":2197},[2195],"language-text","--moe-backend marlin\n","text",[129,2199,2196],{"__ignoreMap":35},[10,2201,2202],{},"Marlin is a CUTLASS-based INT4\u002FFP4 GEMM kernel that's mature and fast on Blackwell.",[2142,2204,2206],{"id":2205},"heterogeneous-head-dimensions","Heterogeneous Head Dimensions",[10,2208,2209,2210,2213],{},"Gemma 4 uses different attention head dimensions for local vs global attention (256 vs 512). vLLM automatically detects this and forces the ",[515,2211,2212],{},"TRITON_ATTN"," backend to avoid mixed-backend numerical divergence. Nothing for us to configure, it just works, but worth understanding why Flash Attention isn't in play here.",[2142,2215,2217],{"id":2216},"kv-cache-memory","KV Cache Memory",[10,2219,2220,2221,2224],{},"We enable ",[129,2222,2223],{},"--kv-cache-dtype fp8"," to halve KV cache memory. With 131K context and 16 concurrent sequences, this makes the difference between fitting in 128GB and OOMing. Quality impact is negligible for our workloads.",[2142,2226,2228],{"id":2227},"docker-runtime-gotcha","Docker Runtime Gotcha",[10,2230,2231,2232,2235,2236,2239,2240,2243,2244,79],{},"On the office Spark, the ",[129,2233,2234],{},"nvidia"," Docker runtime wasn't registered (only the container toolkit was installed). The fix was to use ",[129,2237,2238],{},"--gpus all"," instead of ",[129,2241,2242],{},"--runtime nvidia",", the older and more portable flag. Worth knowing if you see ",[129,2245,2246],{},"unknown or invalid runtime name: nvidia",[93,2248,2250],{"id":2249},"the-full-launch-command","The Full Launch Command",[824,2252,2254],{"className":826,"code":2253,"language":828,"meta":35,"style":35},"docker run -d \\\n  --name vllm-avarok \\\n  --gpus all \\\n  --shm-size=16g \\\n  --restart unless-stopped \\\n  -p 8000:8888 \\\n  -v \u002Fhome\u002F$USER\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface \\\n  -v \u002Fhome\u002F$USER\u002Fvllm\u002Fgemma4_patched.py:\u002Fusr\u002Flocal\u002Flib\u002Fpython3.12\u002Fdist-packages\u002Fvllm\u002Fmodel_executor\u002Fmodels\u002Fgemma4.py:ro \\\n  vllm\u002Fvllm-openai:gemma4-cu130 \\\n  --model bg-digitalservices\u002FGemma-4-26B-A4B-it-NVFP4 \\\n  --served-model-name google\u002Fgemma-4-26B-A4B-it \\\n  --host 0.0.0.0 \\\n  --port 8888 \\\n  --quantization modelopt \\\n  --moe-backend marlin \\\n  --kv-cache-dtype fp8 \\\n  --enable-prefix-caching \\\n  --enable-chunked-prefill \\\n  --max-model-len 131072 \\\n  --gpu-memory-utilization 0.85 \\\n  --max-num-seqs 16 \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser gemma4 \\\n  --reasoning-parser gemma4\n",[129,2255,2256,2266,2275,2283,2289,2297,2305,2317,2330,2337,2347,2357,2367,2377,2387,2398,2409,2417,2425,2436,2447,2458,2466,2477],{"__ignoreMap":35},[832,2257,2258,2260,2262,2264],{"class":834,"line":226},[832,2259,838],{"class":837},[832,2261,858],{"class":841},[832,2263,861],{"class":841},[832,2265,865],{"class":864},[832,2267,2268,2270,2273],{"class":834,"line":36},[832,2269,871],{"class":841},[832,2271,2272],{"class":841}," vllm-avarok",[832,2274,865],{"class":864},[832,2276,2277,2279,2281],{"class":834,"line":853},[832,2278,882],{"class":841},[832,2280,885],{"class":841},[832,2282,865],{"class":864},[832,2284,2285,2287],{"class":834,"line":868},[832,2286,893],{"class":841},[832,2288,865],{"class":864},[832,2290,2291,2293,2295],{"class":834,"line":879},[832,2292,901],{"class":841},[832,2294,904],{"class":841},[832,2296,865],{"class":864},[832,2298,2299,2301,2303],{"class":834,"line":890},[832,2300,912],{"class":841},[832,2302,915],{"class":841},[832,2304,865],{"class":864},[832,2306,2307,2309,2311,2313,2315],{"class":834,"line":898},[832,2308,923],{"class":841},[832,2310,926],{"class":841},[832,2312,929],{"class":864},[832,2314,932],{"class":841},[832,2316,865],{"class":864},[832,2318,2319,2321,2323,2325,2328],{"class":834,"line":909},[832,2320,923],{"class":841},[832,2322,926],{"class":841},[832,2324,929],{"class":864},[832,2326,2327],{"class":841},"\u002Fvllm\u002Fgemma4_patched.py:\u002Fusr\u002Flocal\u002Flib\u002Fpython3.12\u002Fdist-packages\u002Fvllm\u002Fmodel_executor\u002Fmodels\u002Fgemma4.py:ro",[832,2329,865],{"class":864},[832,2331,2332,2335],{"class":834,"line":920},[832,2333,2334],{"class":841},"  vllm\u002Fvllm-openai:gemma4-cu130",[832,2336,865],{"class":864},[832,2338,2339,2342,2345],{"class":834,"line":937},[832,2340,2341],{"class":841},"  --model",[832,2343,2344],{"class":841}," bg-digitalservices\u002FGemma-4-26B-A4B-it-NVFP4",[832,2346,865],{"class":864},[832,2348,2349,2352,2355],{"class":834,"line":948},[832,2350,2351],{"class":841},"  --served-model-name",[832,2353,2354],{"class":841}," google\u002Fgemma-4-26B-A4B-it",[832,2356,865],{"class":864},[832,2358,2359,2362,2365],{"class":834,"line":39},[832,2360,2361],{"class":841},"  --host",[832,2363,2364],{"class":956}," 0.0.0.0",[832,2366,865],{"class":864},[832,2368,2369,2372,2375],{"class":834,"line":974},[832,2370,2371],{"class":841},"  --port",[832,2373,2374],{"class":956}," 8888",[832,2376,865],{"class":864},[832,2378,2379,2382,2385],{"class":834,"line":987},[832,2380,2381],{"class":841},"  --quantization",[832,2383,2384],{"class":841}," modelopt",[832,2386,865],{"class":864},[832,2388,2390,2393,2396],{"class":834,"line":2389},15,[832,2391,2392],{"class":841},"  --moe-backend",[832,2394,2395],{"class":841}," marlin",[832,2397,865],{"class":864},[832,2399,2401,2404,2407],{"class":834,"line":2400},16,[832,2402,2403],{"class":841},"  --kv-cache-dtype",[832,2405,2406],{"class":841}," fp8",[832,2408,865],{"class":864},[832,2410,2412,2415],{"class":834,"line":2411},17,[832,2413,2414],{"class":841},"  --enable-prefix-caching",[832,2416,865],{"class":864},[832,2418,2420,2423],{"class":834,"line":2419},18,[832,2421,2422],{"class":841},"  --enable-chunked-prefill",[832,2424,865],{"class":864},[832,2426,2428,2431,2434],{"class":834,"line":2427},19,[832,2429,2430],{"class":841},"  --max-model-len",[832,2432,2433],{"class":956}," 131072",[832,2435,865],{"class":864},[832,2437,2439,2442,2445],{"class":834,"line":2438},20,[832,2440,2441],{"class":841},"  --gpu-memory-utilization",[832,2443,2444],{"class":956}," 0.85",[832,2446,865],{"class":864},[832,2448,2450,2453,2456],{"class":834,"line":2449},21,[832,2451,2452],{"class":841},"  --max-num-seqs",[832,2454,2455],{"class":956}," 16",[832,2457,865],{"class":864},[832,2459,2461,2464],{"class":834,"line":2460},22,[832,2462,2463],{"class":841},"  --enable-auto-tool-choice",[832,2465,865],{"class":864},[832,2467,2469,2472,2475],{"class":834,"line":2468},23,[832,2470,2471],{"class":841},"  --tool-call-parser",[832,2473,2474],{"class":841}," gemma4",[832,2476,865],{"class":864},[832,2478,2480,2483],{"class":834,"line":2479},24,[832,2481,2482],{"class":841},"  --reasoning-parser",[832,2484,2485],{"class":841}," gemma4\n",[10,2487,2488],{},"A few flags to highlight:",[536,2490,2491,2497,2505,2510,2519],{},[539,2492,2493,2496],{},[129,2494,2495],{},"--served-model-name google\u002Fgemma-4-26B-A4B-it",", clients use this name in the API, regardless of the underlying quantized checkpoint. Makes it easy to swap quantization levels later.",[539,2498,2499,2501,2502,2504],{},[129,2500,1698],{}," + ",[129,2503,1717],{},", native function calling, OpenAI-compatible API.",[539,2506,2507,2509],{},[129,2508,1720],{},", enables the thinking mode output channel.",[539,2511,2512,2501,2515,2518],{},[129,2513,2514],{},"--enable-prefix-caching",[129,2516,2517],{},"--enable-chunked-prefill",", standard vLLM performance wins.",[539,2520,2521,2523],{},[129,2522,1774],{},", we tune this down from the default 128 because our workloads are small-team, not high-concurrency serving. Frees up memory for larger context windows.",[10,2525,2526],{},"First boot takes 10-20 minutes (model download + load + CUDA graph capture). Subsequent restarts take ~3 minutes.",[93,2528,2530],{"id":2529},"what-tool-calling-unlocks","What Tool Calling Unlocks",[10,2532,2533],{},"With Gemma 4 running, we can now pass tool schemas directly to the API:",[824,2535,2539],{"className":2536,"code":2537,"language":2538,"meta":35,"style":35},"language-python shiki shiki-themes min-light","from openai import OpenAI\n\nclient = OpenAI(base_url=\"http:\u002F\u002Fspark:8000\u002Fv1\", api_key=\"unused\")\n\ntools = [{\n    \"type\": \"function\",\n    \"function\": {\n        \"name\": \"get_xrpl_account_balance\",\n        \"description\": \"Fetch the XRP balance for an XRPL account\",\n        \"parameters\": {\n            \"type\": \"object\",\n            \"properties\": {\n                \"address\": {\"type\": \"string\", \"description\": \"Classic XRPL address starting with r\"}\n            },\n            \"required\": [\"address\"]\n        }\n    }\n}]\n\nresponse = client.chat.completions.create(\n    model=\"google\u002Fgemma-4-26B-A4B-it\",\n    messages=[{\"role\": \"user\", \"content\": \"How much XRP does rMCU4... hold?\"}],\n    tools=tools,\n    tool_choice=\"auto\",\n)\n\n# response.choices[0].message.tool_calls -> [ChatCompletionMessageToolCall(...)]\n","python",[129,2540,2541,2556,2560,2592,2596,2609,2623,2633,2645,2657,2666,2678,2687,2719,2724,2739,2744,2749,2756,2760,2788,2800,2831,2841,2853,2858,2863],{"__ignoreMap":35},[832,2542,2543,2547,2550,2553],{"class":834,"line":226},[832,2544,2546],{"class":2545},"s-F7R","from",[832,2548,2549],{"class":864}," openai ",[832,2551,2552],{"class":2545},"import",[832,2554,2555],{"class":864}," OpenAI\n",[832,2557,2558],{"class":834,"line":36},[832,2559,850],{"emptyLinePlaceholder":45},[832,2561,2562,2565,2568,2571,2575,2577,2581,2584,2586,2589],{"class":834,"line":853},[832,2563,2564],{"class":864},"client ",[832,2566,2567],{"class":2545},"=",[832,2569,2570],{"class":837}," OpenAI",[832,2572,2574],{"class":2573},"siqTm","(base_url",[832,2576,2567],{"class":2545},[832,2578,2580],{"class":2579},"shJU0","\"http:\u002F\u002Fspark:8000\u002Fv1\"",[832,2582,2583],{"class":2573},", api_key",[832,2585,2567],{"class":2545},[832,2587,2588],{"class":2579},"\"unused\"",[832,2590,2591],{"class":2573},")\n",[832,2593,2594],{"class":834,"line":868},[832,2595,850],{"emptyLinePlaceholder":45},[832,2597,2598,2601,2603,2606],{"class":834,"line":879},[832,2599,2600],{"class":864},"tools ",[832,2602,2567],{"class":2545},[832,2604,2605],{"class":864}," [",[832,2607,2608],{"class":2573},"{\n",[832,2610,2611,2614,2617,2620],{"class":834,"line":890},[832,2612,2613],{"class":2579},"    \"type\"",[832,2615,2616],{"class":2573},":",[832,2618,2619],{"class":2579}," \"function\"",[832,2621,2622],{"class":2573},",\n",[832,2624,2625,2628,2630],{"class":834,"line":898},[832,2626,2627],{"class":2579},"    \"function\"",[832,2629,2616],{"class":2573},[832,2631,2632],{"class":2573}," {\n",[832,2634,2635,2638,2640,2643],{"class":834,"line":909},[832,2636,2637],{"class":2579},"        \"name\"",[832,2639,2616],{"class":2573},[832,2641,2642],{"class":2579}," \"get_xrpl_account_balance\"",[832,2644,2622],{"class":2573},[832,2646,2647,2650,2652,2655],{"class":834,"line":920},[832,2648,2649],{"class":2579},"        \"description\"",[832,2651,2616],{"class":2573},[832,2653,2654],{"class":2579}," \"Fetch the XRP balance for an XRPL account\"",[832,2656,2622],{"class":2573},[832,2658,2659,2662,2664],{"class":834,"line":937},[832,2660,2661],{"class":2579},"        \"parameters\"",[832,2663,2616],{"class":2573},[832,2665,2632],{"class":2573},[832,2667,2668,2671,2673,2676],{"class":834,"line":948},[832,2669,2670],{"class":2579},"            \"type\"",[832,2672,2616],{"class":2573},[832,2674,2675],{"class":2579}," \"object\"",[832,2677,2622],{"class":2573},[832,2679,2680,2683,2685],{"class":834,"line":39},[832,2681,2682],{"class":2579},"            \"properties\"",[832,2684,2616],{"class":2573},[832,2686,2632],{"class":2573},[832,2688,2689,2692,2694,2697,2700,2702,2705,2708,2711,2713,2716],{"class":834,"line":974},[832,2690,2691],{"class":2579},"                \"address\"",[832,2693,2616],{"class":2573},[832,2695,2696],{"class":2573}," {",[832,2698,2699],{"class":2579},"\"type\"",[832,2701,2616],{"class":2573},[832,2703,2704],{"class":2579}," \"string\"",[832,2706,2707],{"class":2573},",",[832,2709,2710],{"class":2579}," \"description\"",[832,2712,2616],{"class":2573},[832,2714,2715],{"class":2579}," \"Classic XRPL address starting with r\"",[832,2717,2718],{"class":2573},"}\n",[832,2720,2721],{"class":834,"line":987},[832,2722,2723],{"class":2573},"            },\n",[832,2725,2726,2729,2731,2733,2736],{"class":834,"line":2389},[832,2727,2728],{"class":2579},"            \"required\"",[832,2730,2616],{"class":2573},[832,2732,2605],{"class":864},[832,2734,2735],{"class":2579},"\"address\"",[832,2737,2738],{"class":864},"]\n",[832,2740,2741],{"class":834,"line":2400},[832,2742,2743],{"class":2573},"        }\n",[832,2745,2746],{"class":834,"line":2411},[832,2747,2748],{"class":2573},"    }\n",[832,2750,2751,2754],{"class":834,"line":2419},[832,2752,2753],{"class":2573},"}",[832,2755,2738],{"class":864},[832,2757,2758],{"class":834,"line":2427},[832,2759,850],{"emptyLinePlaceholder":45},[832,2761,2762,2765,2767,2770,2772,2775,2777,2780,2782,2785],{"class":834,"line":2438},[832,2763,2764],{"class":864},"response ",[832,2766,2567],{"class":2545},[832,2768,2769],{"class":864}," client",[832,2771,79],{"class":2573},[832,2773,2774],{"class":864},"chat",[832,2776,79],{"class":2573},[832,2778,2779],{"class":864},"completions",[832,2781,79],{"class":2573},[832,2783,2784],{"class":837},"create",[832,2786,2787],{"class":2573},"(\n",[832,2789,2790,2793,2795,2798],{"class":834,"line":2449},[832,2791,2792],{"class":2573},"    model",[832,2794,2567],{"class":2545},[832,2796,2797],{"class":2579},"\"google\u002Fgemma-4-26B-A4B-it\"",[832,2799,2622],{"class":2573},[832,2801,2802,2805,2807,2810,2813,2815,2818,2820,2823,2825,2828],{"class":834,"line":2460},[832,2803,2804],{"class":2573},"    messages",[832,2806,2567],{"class":2545},[832,2808,2809],{"class":2573},"[{",[832,2811,2812],{"class":2579},"\"role\"",[832,2814,1519],{"class":2573},[832,2816,2817],{"class":2579},"\"user\"",[832,2819,1523],{"class":2573},[832,2821,2822],{"class":2579},"\"content\"",[832,2824,1519],{"class":2573},[832,2826,2827],{"class":2579},"\"How much XRP does rMCU4... hold?\"",[832,2829,2830],{"class":2573},"}],\n",[832,2832,2833,2836,2838],{"class":834,"line":2468},[832,2834,2835],{"class":2573},"    tools",[832,2837,2567],{"class":2545},[832,2839,2840],{"class":2573},"tools,\n",[832,2842,2843,2846,2848,2851],{"class":834,"line":2479},[832,2844,2845],{"class":2573},"    tool_choice",[832,2847,2567],{"class":2545},[832,2849,2850],{"class":2579},"\"auto\"",[832,2852,2622],{"class":2573},[832,2854,2856],{"class":834,"line":2855},25,[832,2857,2591],{"class":2573},[832,2859,2861],{"class":834,"line":2860},26,[832,2862,850],{"emptyLinePlaceholder":45},[832,2864,2866],{"class":834,"line":2865},27,[832,2867,2869],{"class":2868},"s15Vz","# response.choices[0].message.tool_calls -> [ChatCompletionMessageToolCall(...)]\n",[10,2871,2872],{},"This is the foundation for agentic workflows that previously required prompt engineering or separate function-calling layers (like LiteLLM's function-call emulation). With Gemma 4, the model speaks tool calls natively.",[10,2874,2875,2876,2879],{},"The reasoning mode is similarly useful. For complex queries, the model emits a private thinking trace before its final answer, we can log it for debugging, show it to users for transparency, or strip it entirely. All via the ",[129,2877,2878],{},"--reasoning-parser"," flag.",[93,2881,2883],{"id":2882},"the-benchmarks","The Benchmarks",[10,2885,2886],{},"We ran a proper benchmark suite from the Spark itself (not over the network, to remove noise): throughput at 100\u002F500\u002F2000-token outputs, time-to-first-token, concurrency scaling, prefix-cache hit\u002Fmiss, long-context prefill, and tool-calling overhead.",[2142,2888,2890],{"id":2889},"single-stream-throughput","Single-stream throughput",[712,2892,2893,2903],{},[715,2894,2895],{},[718,2896,2897,2900],{},[721,2898,2899],{},"Generation length",[721,2901,2902],{},"tok\u002Fs",[734,2904,2905,2913,2923],{},[718,2906,2907,2910],{},[739,2908,2909],{},"100 tokens",[739,2911,2912],{},"46",[718,2914,2915,2918],{},[739,2916,2917],{},"500 tokens",[739,2919,2920],{},[515,2921,2922],{},"51",[718,2924,2925,2928],{},[739,2926,2927],{},"2000 tokens",[739,2929,2930],{},"50",[10,2932,2933],{},"The short-generation number is lower only because warmup dominates. Anything over ~200 tokens hits the steady-state ~50 tok\u002Fs. That matches the theoretical ceiling for 4B active parameters at FP4 on 273 GB\u002Fs bandwidth.",[2142,2935,2937],{"id":2936},"concurrency-the-big-win-for-agentic-workflows","Concurrency, the big win for agentic workflows",[10,2939,2940],{},"This is where the Spark shines. vLLM's continuous batching + Marlin NVFP4 MoE kernel keeps per-request throughput high even as you add parallel clients:",[712,2942,2943,2956],{},[715,2944,2945],{},[718,2946,2947,2950,2953],{},[721,2948,2949],{},"Parallel requests",[721,2951,2952],{},"Aggregate tok\u002Fs",[721,2954,2955],{},"Per-request tok\u002Fs",[734,2957,2958,2967,2978,2989],{},[718,2959,2960,2963,2965],{},[739,2961,2962],{},"1",[739,2964,2930],{},[739,2966,2930],{},[718,2968,2969,2972,2975],{},[739,2970,2971],{},"2",[739,2973,2974],{},"90",[739,2976,2977],{},"45",[718,2979,2980,2983,2986],{},[739,2981,2982],{},"4",[739,2984,2985],{},"161",[739,2987,2988],{},"40",[718,2990,2991,2996,3001],{},[739,2992,2993],{},[515,2994,2995],{},"8",[739,2997,2998],{},[515,2999,3000],{},"271",[739,3002,3003],{},"34",[10,3005,3006],{},"5.4× aggregate throughput at 8 concurrent clients. For a team of devs running agents in parallel, or a single agent making parallel tool-call decisions, this is huge.",[2142,3008,3010],{"id":3009},"prefix-caching-massive-for-agent-loops","Prefix caching, massive for agent loops",[10,3012,3013],{},"Agent workflows resubmit the same system prompt, same tool schema, same conversation history over and over. Measured on an 8K-token prefix:",[712,3015,3016,3025],{},[715,3017,3018],{},[718,3019,3020,3022],{},[721,3021],{},[721,3023,3024],{},"TTFT",[734,3026,3027,3035],{},[718,3028,3029,3032],{},[739,3030,3031],{},"Cold (cache miss)",[739,3033,3034],{},"834 ms",[718,3036,3037,3040],{},[739,3038,3039],{},"Warm (cache hit)",[739,3041,3042],{},[515,3043,3044],{},"102 ms",[10,3046,3047,3050],{},[515,3048,3049],{},"8.2× speedup."," If you're building anything where the same prefix repeats, ReAct loops, multi-turn conversations, a shared system prompt, prefix caching alone pays for running local inference.",[2142,3052,3054],{"id":3053},"ttft-and-long-context","TTFT and long context",[536,3056,3057,3063,3066,3069,3075],{},[539,3058,3059,3060,3062],{},"Short prompt TTFT: ",[515,3061,1986],{}," p50 (excellent)",[539,3064,3065],{},"4K prompt TTFT: 62 ms p50",[539,3067,3068],{},"8K prefill: 73 ms",[539,3070,3071,3072],{},"32K prefill: ",[515,3073,3074],{},"10.1 seconds",[539,3076,3077,3078],{},"100K prefill: ",[515,3079,3080],{},"78 seconds",[10,3082,3083],{},"Long-context prefill is the one real weakness. It's compute-bound on the MoE GEMM kernel, not bandwidth-bound. At 131K context you're paying 80+ seconds of prefill time every cold turn. If you need long-context frequently, design your agents to reuse prefixes so the cache does the heavy lifting.",[2142,3085,3087],{"id":3086},"tool-calling","Tool calling",[10,3089,3090,3091,3094],{},"Works out of the box with ",[129,3092,3093],{},"--enable-auto-tool-choice --tool-call-parser gemma4",". No meaningful overhead versus text-only completion, the model just emits shorter, structured output when it decides to call a tool.",[93,3096,3098],{"id":3097},"trying-to-push-throughput-further","Trying to Push Throughput Further",[10,3100,3101],{},"With the baseline measured, we tried four paths to push single-stream throughput higher. All four hit walls:",[712,3103,3104,3114],{},[715,3105,3106],{},[718,3107,3108,3111],{},[721,3109,3110],{},"Attempt",[721,3112,3113],{},"Result",[734,3115,3116,3127,3137,3153],{},[718,3117,3118,3124],{},[739,3119,3120,3121],{},"Swap MoE backend to ",[129,3122,3123],{},"flashinfer_trtllm",[739,3125,3126],{},"Kernel doesn't support SM 12.1, engine fails to start",[718,3128,3129,3134],{},[739,3130,3120,3131],{},[129,3132,3133],{},"flashinfer_cutlass",[739,3135,3136],{},"Doesn't support GELU activation (Gemma 4 uses GELU)",[718,3138,3139,3150],{},[739,3140,3141,3142,3145,3146,3149],{},"Bump ",[129,3143,3144],{},"--gpu-memory-utilization"," 0.85 → 0.90 + ",[129,3147,3148],{},"--max-num-seqs"," 16 → 32",[739,3151,3152],{},"Zero measurable change, we're not KV-cache-limited at this scale",[718,3154,3155,3158],{},[739,3156,3157],{},"Speculative decoding with Gemma-4-E4B draft",[739,3159,3160],{},"Blocked, vLLM's spec decoding doesn't support multimodal target models",[10,3162,3163],{},"The last one hurt. Spec decoding was the one real lever: a small 4B draft model predicting tokens for the 26B target can deliver 1.5-2× throughput on repetitive outputs. But vLLM explicitly rejects it for multimodal models right now, and Gemma 4 is multimodal. The two features we most wanted, speculative decoding and image input, are mutually exclusive in today's vLLM.",[10,3165,3166,3169],{},[515,3167,3168],{},"Conclusion: the baseline config is already at the hardware + software ceiling."," Generation throughput is memory-bandwidth-bound. Prefill is compute-bound on the only NVFP4 MoE kernel that works on GB10 + Gemma 4 (Marlin). There's no software knob we haven't turned.",[93,3171,3173],{"id":3172},"what-would-change-the-picture","What Would Change the Picture",[10,3175,3176],{},"Future improvements we'll re-test when they land:",[536,3178,3179,3185,3191,3197],{},[539,3180,3181,3184],{},[515,3182,3183],{},"vLLM adds multimodal spec decoding",", unlocks Gemma-4-E4B as draft, expected 1.5-2×",[539,3186,3187,3190],{},[515,3188,3189],{},"FlashInfer adds SM 12.1 + GELU support",", alternative MoE backend, could improve prefill",[539,3192,3193,3196],{},[515,3194,3195],{},"Community ships a clean Gemma 4 AWQ INT4 quant",", Qwen AWQ hit 82 tok\u002Fs on the same hardware",[539,3198,3199,3202],{},[515,3200,3201],{},"NVIDIA publishes an official NVFP4 Gemma 4 build",", likely has better-tuned kernels",[10,3204,3205,3206,3209],{},"Plan: re-run the benchmark suite (checked in at ",[129,3207,3208],{},"projects\u002Fdgx-spark-benchmarks\u002Fbench.py",") every couple of months and see if the landscape has shifted.",[93,3211,3213],{"id":3212},"whats-next-for-us","What's Next for Us",[536,3215,3216,3222,3237],{},[539,3217,3218,3221],{},[515,3219,3220],{},"Multimodal testing",", Gemma 4 accepts image inputs. Document parsing and UI automation are the obvious use cases.",[539,3223,3224,1523,3227,3230,3231,3236],{},[515,3225,3226],{},"Local speech-to-text",[129,3228,3229],{},"faster-whisper"," on the Grace CPU cores, leaving the GPU fully dedicated to Gemma 4. ",[675,3232,3235],{"href":3233,"rel":3234},"https:\u002F\u002Flearn.arm.com\u002Flearning-paths\u002Flaptops-and-desktops\u002Fdgx_spark_voicechatbot\u002F",[679],"Arm published a playbook"," for exactly this split on the Spark.",[539,3238,3239,3242],{},[515,3240,3241],{},"Agent workload benchmarks",", beyond synthetic throughput, measure actual tool-call accuracy and reasoning quality on our real XRPL workflows.",[93,3244,1064],{"id":1063},[536,3246,3247,3254,3260,3267],{},[539,3248,3249],{},[675,3250,3253],{"href":3251,"rel":3252},"https:\u002F\u002Fhuggingface.co\u002Fblog\u002Fgemma4",[679],"Gemma 4 on HuggingFace",[539,3255,3256],{},[675,3257,2114],{"href":3258,"rel":3259},"https:\u002F\u002Fhuggingface.co\u002Fbg-digitalservices\u002FGemma-4-26B-A4B-it-NVFP4",[679],[539,3261,3262],{},[675,3263,3266],{"href":3264,"rel":3265},"https:\u002F\u002Fdocs.vllm.ai\u002Fprojects\u002Frecipes\u002Fen\u002Flatest\u002FGoogle\u002FGemma4.html",[679],"vLLM Gemma 4 recipe",[539,3268,3269],{},[675,3270,3271],{"href":2007},"Our original DGX Spark setup post",[93,3273,1097],{"id":1096},[10,3275,3276,3277,3279,3280,3283,3284,3286,3287,3290],{},"If you have a DGX Spark and want to run Gemma 4: pull the ",[129,3278,2135],{}," image, get a working ",[129,3281,3282],{},"gemma4_patched.py",", and use the ",[129,3285,705],{}," command above. Allow ~20 minutes for the first boot. Reach out at ",[675,3288,1105],{"href":1103,"rel":3289},[679]," if you hit issues, we've documented most of the sharp edges.",[1108,3292,3293],{},"html pre.shiki code .s7eDp, html code.shiki .s7eDp{--shiki-default:#6F42C1}html pre.shiki code .sY4mW, html code.shiki .sY4mW{--shiki-default:#2B5581}html pre.shiki code .sR6ew, html code.shiki .sR6ew{--shiki-default:#24292EFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html pre.shiki code .s9AOD, html code.shiki .s9AOD{--shiki-default:#1976D2}html pre.shiki code .s-F7R, html code.shiki .s-F7R{--shiki-default:#D32F2F}html pre.shiki code .siqTm, html code.shiki .siqTm{--shiki-default:#212121}html pre.shiki code .shJU0, html code.shiki .shJU0{--shiki-default:#22863A}html pre.shiki code .s15Vz, html code.shiki .s15Vz{--shiki-default:#C2C3C5}",{"title":35,"searchDepth":36,"depth":36,"links":3295},[3296,3297,3298,3299,3306,3307,3308,3315,3316,3317,3318,3319],{"id":2001,"depth":36,"text":2002},{"id":2040,"depth":36,"text":2041},{"id":2107,"depth":36,"text":2108},{"id":2139,"depth":36,"text":2140,"children":3300},[3301,3302,3303,3304,3305],{"id":2144,"depth":853,"text":2145},{"id":2177,"depth":853,"text":2178},{"id":2205,"depth":853,"text":2206},{"id":2216,"depth":853,"text":2217},{"id":2227,"depth":853,"text":2228},{"id":2249,"depth":36,"text":2250},{"id":2529,"depth":36,"text":2530},{"id":2882,"depth":36,"text":2883,"children":3309},[3310,3311,3312,3313,3314],{"id":2889,"depth":853,"text":2890},{"id":2936,"depth":853,"text":2937},{"id":3009,"depth":853,"text":3010},{"id":3053,"depth":853,"text":3054},{"id":3086,"depth":853,"text":3087},{"id":3097,"depth":36,"text":3098},{"id":3172,"depth":36,"text":3173},{"id":3212,"depth":36,"text":3213},{"id":1063,"depth":36,"text":1064},{"id":1096,"depth":36,"text":1097},"2026-04-22","TL;DR: We set up Google's Gemma 4 26B-A4B NVFP4 on the XRPL Commons office DGX Spark, with native tool calling, reasoning mode, 131K context, and multimodal input preserved on 128GB of unified memory. We then tried to push throughput further and hit the hardware's real limits. Here's the full story.",{},"\u002Fideas\u002F2026-04-22-dgx-spark-gemma4",{"title":1955,"description":3321},"ideas\u002F2026-04-22-dgx-spark-gemma4","Native tool calling and reasoning mode on Gemma 4 NVFP4 over 128GB of unified memory.",[1133,1134,1135,3328],"gemma","HJg1bpC4WTA55XH45oFSwAvD6I-PF3WTHHwLgP53Erc",{"id":3331,"title":3332,"body":3333,"canonical_url":38,"chapter":920,"date":3370,"description":3339,"extension":41,"featured":42,"image":38,"kind":43,"meta":3371,"navigation":45,"path":3372,"seo":3373,"series":48,"stem":3374,"summary":3375,"tags":3376,"work_slug":52,"__hash__":3377},"writing\u002Fideas\u002F2026-06-05-anton-09-changing-my-mind.md","Anton, chapter 9: Changing my mind about home",{"type":7,"value":3334,"toc":3368},[3335,3340,3343,3350,3353,3356,3359,3362,3365],[10,3336,3337],{},[13,3338,3339],{},"2026-04-29 → 2026-06-05",[10,3341,3342],{},"The Spark goes quiet one day in May, and stays quiet. Not a crash, nothing to debug remotely: the box at home is simply unreachable, for days, while I'm not there to press a button. And the thing I notice isn't panic. It's that the family barely notices, because the cloud fallback carries the conversations, and what actually breaks is everything else: the scheduled jobs, the memory writes, the heartbeat, the whole household layer that lives on the same machine as the model.",[10,3344,3345,3346,3349],{},"This forces the conversation I've been avoiding with myself since chapter 1. The founding instinct was that Anton lives at home, on my hardware, close to the family. It's a good instinct and I still believe the half of it that matters. But a home machine has home problems: it shares the power, the network, and the physical room with a family of six, and it runs hot under sustained load in a way a datacenter never lets you feel. The realization, once I stop defending the original position, is that I'd bundled two different claims into one. The model should run on my hardware, because that's where the privacy and the economics point. The ",[13,3347,3348],{},"system"," (the always-on plumbing that the family depends on at 7am) should run wherever it's most boring and most reachable. Those were never the same requirement. I'd just wanted them to be.",[10,3351,3352],{},"So the stack moves. The worker, the bridges, the database, the whole application layer goes to a small rented server, always on, reachable from anywhere over the tailnet. The Spark keeps the one job it is uniquely good at: serving the model. It stops being the host and becomes what it functionally always was, a local inference endpoint, no different in shape from a cloud provider except that it's mine. Sometimes you change your mind in a project, and the honest version of this story is that the architecture didn't fail; my framing of it did.",[10,3354,3355],{},"The move surfaces the debt that hiding at home had let accumulate. One session chasing \"I should be seeing new episodes and I don't\" turns up six independent bugs stacked end to end in the media pipeline, each one masking the next: an air-date filter rejecting every torrent named by episode, a download budget that starved the back half of the watchlist forever, a skill that couldn't reach the download host because it never declared the secret it needed, and three more behind those. Nothing in the stack had been lying loudly. Everything had been failing quietly, in layers, and looking healthy at a glance. A pipeline you don't measure end to end isn't a pipeline, it's a hope.",[10,3357,3358],{},"The next problem pulls me deeper. Investigating why the local model keeps being unreliable as Anton's brain, I finally measure the one thing I'd never measured: the prompt. Eleven thousand four hundred tokens, median, every round. Identity, a storage decision tree, a cast table, thread-awareness rules, auto-loaded memory, auto-loaded directives, auto-loaded recent actions, all of it stacked in front of every single turn. A quarter of the local model's context spent before any work begins, attention diluted to nothing by round three. The diagnosis is uncomfortable: the architecture was written for a frontier model's attention span and then handed to a small one. The model was never the problem. I was asking it to read a novel before every sentence.",[10,3360,3361],{},"The fix is the deepest cut since the framework came out in chapter 5. Anton becomes a router. A three-hundred-token system prompt whose only job is picking the right specialist and composing the reply. The auto-loaded everything is gone: memory, directives, and recent actions stop being ambient context and become things the agent calls when the question needs them. Ask what time it is, and no memory is fetched, no facts ride along, nothing is spent. Ask about Clara's shoe size, and the memory specialist gets called and returns exactly that. The specialists own their domains behind prompts capped small enough that a small model can actually hold them. It's the chapter 4 lesson (give the LLM a small menu) applied to the one agent I'd exempted from it: the one at the top.",[10,3363,3364],{},"The same discipline turns destructive, in the good way, on my monitoring system. I'd built a signals primitive over the spring: definitions table, time-series, stream, drainer, breach state machine, remedy plans. Reviewing it after a false alarm, I make the call to delete nearly all of it. What a one-operator system actually needs to answer \"are you healthy?\" is a cron that writes a few metric rows and one skill that reads them. Build the abstraction, watch it meet the system's real scale, and keep removing until what's left is exactly what earns its keep. The best architecture review is the one where most of the architecture doesn't survive.",[10,3366,3367],{},"By the time this stretch of work settles, the shape is honest for the first time. The family layer runs where uptime is boring. The model runs where privacy is physical. And the system gets smaller twice, once at the top of the agent and once in its nervous system, both times by admitting that something I'd built with conviction was more than the problem ever asked for.",{"title":35,"searchDepth":36,"depth":36,"links":3369},[],"2026-06-05",{},"\u002Fideas\u002F2026-06-05-anton-09-changing-my-mind",{"title":3332,"description":3339},"ideas\u002F2026-06-05-anton-09-changing-my-mind","Moving the application layer to a rented server while keeping the model on hardware at home, admitting the founding architecture needed reframing.",[52,53,54,1134],"d09iw2MVadLJZNJeBMK7fpIVvD6c-Qd7Ex4BnLG4P-8",{"id":3379,"title":3380,"body":3381,"canonical_url":38,"chapter":937,"date":3408,"description":3387,"extension":41,"featured":42,"image":38,"kind":43,"meta":3409,"navigation":45,"path":3410,"seo":3411,"series":48,"stem":3412,"summary":3413,"tags":3414,"work_slug":52,"__hash__":3415},"writing\u002Fideas\u002F2026-06-06-anton-10-making-gemma-fly.md","Anton, chapter 10: Making Gemma fly",{"type":7,"value":3382,"toc":3406},[3383,3388,3391,3394,3397,3400,3403],[10,3384,3385],{},[13,3386,3387],{},"2026-06-06 → 2026-06-07",[10,3389,3390],{},"The local model on the Spark is Nemotron when this starts: a reasoning model from one vendor, around fifty-eight tokens a second, held together by two unmerged patches I'd backported by hand so the image would even load it. It's the third occupant of the slot since spring (the Gemma from chapter 7 gave way to a Qwen release, which gave way to this), each swap chasing the next promising checkpoint. Nemotron ships no drafter and no speculative decoding; the whole bet was that distillation-time accuracy would carry it. With a fresh wave of Gemma 4 releases out, multimodal and ours to shape, I move the local model back to it, and it settles into the high forties. That should read as progress. Instead it nags at the line I left open in the spring: speculative decoding, a 1.5x I never cashed in. The question sharpens: is the high forties the ceiling of this hardware, or just the ceiling of what I bothered to try? I decide to find out properly, with numbers, and not stop at the first comfortable answer.",[10,3392,3393],{},"The obvious lever is a better quantization, and there's a fresh one to chase: a quantization-aware-training release, weights trained to survive 4-bit. The theory is seductive, full quality at a quarter of the memory. I fan out a small army of research agents and they come back with the same verdict from five angles: it does nothing for us. The Spark is memory-bandwidth-bound, and any 4-bit format moves the same bytes per token; the number that decides everything is bandwidth, and no checkpoint changes it. Worse, the format the box already runs gets its storage win but not its compute win, because this silicon has no native 4-bit math. A second, identical box has joined the network by now, and benchmarking the same checkpoint on both hands me the sharpest version of the point: one serving image does 52 tokens a second where another does 30, on the same hardware, because only one of them ships the right kernels. The image, not the silicon, sets the floor. I'm carrying the cost of the format (a patched model file, a software-emulated kernel, an image that loads it only if you hold your mouth right) and collecting none of the payoff. Quantization was never the lever.",[10,3395,3396],{},"So I look at the machines I already own. My work Mac and an idle laptop both have more memory bandwidth than the Spark, and decode is bandwidth-bound, so on paper they should win. They lose, badly: the same model runs slower on twice the bandwidth, because Apple's mixture-of-experts kernels pull a third of what the datacenter stack does. Bandwidth sets the ceiling; the kernel decides where you land under it. That detour cost me a clean hypothesis and bought a real rule: the spec sheet is not the benchmark.",[10,3398,3399],{},"The lever turns out to be the one I'd left open. Multi-token prediction: a tiny drafter guesses the next few tokens, the big model verifies them in a single pass, and on a bandwidth-bound box that is the only trick that beats the wall, because you pay for one weight read and get several tokens back. A community write-up claims triple the baseline on this exact hardware, and I want it. I match the recipe to the letter and land at half the headline. The gap is not a missing flag; I run their command verbatim and my baseline matches theirs to the decimal. The gap is what \"tokens per second\" even means. Their number is a harness measuring pure decode with the prefill stripped away; mine is what a caller actually waits for, wall clock, everything included. Both are true. They measure different things. The honest figure, the one a person feels, is a clean 1.4x to 1.9x depending on how predictable the text is, and I write that one down instead of the prettier one.",[10,3401,3402],{},"Then I test it the way Anton is really used: the actual router prompt, a real conversation, sixteen of them at once. The number that matters isn't single-stream at all. When someone is mid-conversation the whole prefix is already cached, and the answer comes back in half a second instead of five, ten times faster on the most common path there is. The win was never a single thing. It was two orthogonal levers, speculation and caching, neither of which has anything to do with the quantization I'd spent days circling.",[10,3404,3405],{},"Last, I try to break it. Five hundred requests at once, and it just queues, never dropping one, the chip barely warm; the old warning about thermal shutdowns at high concurrency refuses to reproduce under a burst, though I don't kid myself that a few minutes of burst is the same as hours of sustained batch on a box that lives in a warm room. So this is where it lands: the local model answers a live conversation in half a second, holds north of a thousand tokens a second across a crowd, and fails over to a second box if the first ever stumbles. The question I started with has an answer. The ceiling I'd accepted was the ceiling of one idea, and most of the climb past it was just measuring honestly enough to see where the floor actually was.",{"title":35,"searchDepth":36,"depth":36,"links":3407},[],"2026-06-06",{},"\u002Fideas\u002F2026-06-06-anton-10-making-gemma-fly",{"title":3380,"description":3387},"ideas\u002F2026-06-06-anton-10-making-gemma-fly","Chasing the local model's performance ceiling with multi-token prediction, prefix caching, and honest benchmarks.",[52,53,54,1133,1866],"XBXs3UEn6dpmhhZIUaP_Zq29OBL6oExUz-bHVb0vFjw",{"id":3417,"title":3418,"body":3419,"canonical_url":38,"chapter":948,"date":3446,"description":3425,"extension":41,"featured":42,"image":38,"kind":43,"meta":3447,"navigation":45,"path":3448,"seo":3449,"series":48,"stem":3450,"summary":3451,"tags":3452,"work_slug":52,"__hash__":3454},"writing\u002Fideas\u002F2026-06-26-anton-11-loop-closes.md","Anton, chapter 11: The loop closes on itself",{"type":7,"value":3420,"toc":3444},[3421,3426,3429,3432,3435,3438,3441],[10,3422,3423],{},[13,3424,3425],{},"2026-06-26 → 2026-07-16",[10,3427,3428],{},"The model flies now, and I still ship every fix by hand. That is the itch I keep returning to. A change lands in the repo, tests go green, and then it sits there until I open a terminal, pull on the home server, rebuild, and watch the health check. The intelligence got fast; the delivery stayed manual. So the question I want to answer is whether the loop can close all the way (issue, review, merge, deploy, running on the family's phones) with nobody in the middle, and without ever shipping something broken to the five people who actually depend on it. The safety half is the whole point. A pipeline that deploys fast and deploys wrong is worse than my hands.",[10,3430,3431],{},"The spine goes in first: a CI gate that will not let anything merge until typecheck, build, and the full test suite are green, and a deploy job that reaches the home server over the private tailnet and runs the health-gated update script I already trust (pull, build, probe health, roll the image back on its own if the probe fails). Keeping the deploy path inside the tailnet means production never opens an SSH door to the public internet; the runner joins the network the same way a laptop would. That part is the boring kind of plumbing that either works or embarrasses you, and it works. (Almost all of it: the deploy notifications turn out to be wired to a channel whose credentials were never set, reporting nothing to no one until I reroute them. Even the messenger needed a health check.)",[10,3433,3434],{},"The smoke gate is where I get educated. The idea is small: after a deploy, ask the live system a handful of questions and only keep the new version if it answers them. The execution fights me for three weeks. First the gate calls an endpoint that isn't there, reads the 404 as a failure, and rolls back every single deploy while paging me CRITICAL. I fix the endpoint and it rolls back again, this time because one of my own test cases asserts that a conversation-search query should route to the knowledge domain when it correctly routes to memory: the gate was wrong, the system was right, and the gate won anyway. I fix the assertion and hit the third wall, the quiet one: six model calls take longer than the timeout, so the gate is silently skipped on every deploy. It has been green the whole time because it never ran. That is the worst failure mode there is. A gate that false-fails gets angry attention and gets fixed; a gate that silently skips reports confidence it never earned, and you find out the day it was supposed to save you.",[10,3436,3437],{},"So I make it fit its own budget. The cases run concurrently instead of one after another, I trim the set until it returns a real verdict inside the window, and then I split the checks into two tiers so the trimmed-back cases come home without power to block: a hard tier that can roll a deploy back, and a soft tier that reports but doesn't. The gate earns its keep for the first time a few days later, when it refuses a commit that genuinely broke two family-facing paths and rolls it back on its own. I didn't catch that one. The thing I built to not trust me caught it, and that is exactly the day the work paid for itself.",[10,3439,3440],{},"The last piece is the fleet that does the building. The sessions that pick up issues and review pull requests are long-lived, and after a full day their context saturates and they start getting slow and strange. So there's a watchdog that notices a stuck session and kills and respawns it with a fresh head, and a supervisor that sleeps the whole fleet when the issue queue is empty and wakes it the moment something new is marked ready. It is a small piece of stagecraft, but it's the difference between an automation that runs for an afternoon and one that runs for weeks.",[10,3442,3443],{},"Where it lands is quieter than I expected. I file an issue, describe the fix I want, and walk away. Some hours later the change is live on the family's phones, or it isn't and nothing shipped, and either way I didn't touch the server. The loop closes on itself. What I notice isn't the speed. It's that the piece I trust most in the whole pipeline is the one whose only job is to stop it.",{"title":35,"searchDepth":36,"depth":36,"links":3445},[],"2026-06-26",{},"\u002Fideas\u002F2026-06-26-anton-11-loop-closes",{"title":3418,"description":3425},"ideas\u002F2026-06-26-anton-11-loop-closes","Building a full CI\u002FCD pipeline with safety gates that can roll back broken deploys, and learning that a gate that silently skips is worse than no gate at all.",[52,53,54,3453,56],"ci-cd","AEM_gnbISU-1PF2KMjLo0FTKyu5pub3MkXH0Eyv4pgI",{"id":4,"title":5,"body":3456,"canonical_url":38,"chapter":39,"date":40,"description":15,"extension":41,"featured":42,"image":38,"kind":43,"meta":3476,"navigation":45,"path":46,"seo":3477,"series":48,"stem":49,"summary":50,"tags":3478,"work_slug":52,"__hash__":57},{"type":7,"value":3457,"toc":3474},[3458,3462,3464,3466,3468,3470,3472],[10,3459,3460],{},[13,3461,15],{},[10,3463,18],{},[10,3465,21],{},[10,3467,24],{},[10,3469,27],{},[10,3471,30],{},[10,3473,33],{},{"title":35,"searchDepth":36,"depth":36,"links":3475},[],{},{"title":5,"description":15},[52,53,54,55,56],{"id":3480,"title":3481,"body":3482,"canonical_url":38,"chapter":38,"date":40,"description":3486,"extension":41,"featured":45,"image":38,"kind":1664,"meta":3567,"navigation":45,"path":3568,"seo":3569,"series":38,"stem":3570,"summary":3571,"tags":3572,"work_slug":38,"__hash__":3575},"writing\u002Fideas\u002F2026-07-14-bring-your-own-ai.md","Bring Your Own AI",{"type":7,"value":3483,"toc":3565},[3484,3487,3490,3493,3496,3504,3507,3510,3513,3516,3519,3522,3525,3528,3531,3534,3537,3540,3543,3550,3553,3556,3559,3562],[10,3485,3486],{},"Whenever we build a product now, we ask the same question: where could AI make this experience easier?",[10,3488,3489],{},"Replace a form with a conversation or let someone upload documents instead. Translate existing material in any language. Query data using natural language.",[10,3491,3492],{},"We've built several of these experiences at Commons. They work. Often incredibly well.",[10,3494,3495],{},"One design pattern has proven especially effective: instead of letting the model generate SQL, API calls, or business logic directly, we define a small domain-specific language.",[10,3497,3498,3501],{},[515,3499,3500],{},"The application owns the semantics.",[515,3502,3503],{},"The model translates the user's intent into that language.",[10,3505,3506],{},"The AI doesn't need to understand the entire system. It just needs to speak the application's language.",[10,3508,3509],{},"That separation of concerns makes the system more reliable, easier to evolve, and much simpler to reason about.",[10,3511,3512],{},"One downside of these new experiences is that every AI feature becomes its own AI product. You're suddenly managing providers, API keys, monitoring, costs, prompts, and ever-changing model capabilities. The rest of your application wants stability; the AI stack refuses to stand still.",[10,3514,3515],{},"This got me thinking.",[10,3517,3518],{},"If the application already owns the semantics, why does it also need to own the intelligence? What if applications simply welcomed agents as first-class users?",[10,3520,3521],{},"Give people secure agent tokens with exactly the permissions they already have. Expose the application's capabilities through a clean API. Whether the interface is MCP or something else almost doesn't matter.",[10,3523,3524],{},"Now users can bring their own AI.",[10,3526,3527],{},"The application still has a great interface. But Claude, ChatGPT, Gemini, or whatever comes next, can also interact with it on the user's behalf.",[10,3529,3530],{},"That AI already knows how the user writes. It speaks their language. It has access to the files they chose to share. It can combine information across several applications instead of being trapped inside one carefully designed chatbot.",[10,3532,3533],{},"The application becomes stable.",[10,3535,3536],{},"The intelligence becomes portable.",[10,3538,3539],{},"There is a loose parallel with identity. An application doesn't necessarily need to create and own your identity; it needs a reliable way to establish who you are and what you're allowed to do.",[10,3541,3542],{},"Perhaps intelligence will become similarly portable.",[10,3544,3545,3546,3549],{},"Instead of every product attempting to own its own AI experience, products may simply become places where ",[515,3547,3548],{},"your"," AI gets work done.",[10,3551,3552],{},"We've started calling this Bring Your Own AI, and we're gradually introducing it across our applications.",[10,3554,3555],{},"Maybe everyone will keep embedding chatbots forever.",[10,3557,3558],{},"Or perhaps, in a few years, building a separate AI into every product will feel as unnecessary as every website implementing its own login system.",[10,3560,3561],{},"I don't know.",[10,3563,3564],{},"I just know this direction feels surprisingly liberating.",{"title":35,"searchDepth":36,"depth":36,"links":3566},[],{},"\u002Fideas\u002F2026-07-14-bring-your-own-ai",{"title":3481,"description":3486},"ideas\u002F2026-07-14-bring-your-own-ai","If the application owns the semantics, why does it also need to own the intelligence? Welcome agents as first-class users: the application becomes stable, the intelligence becomes portable.",[53,54,3573,3574],"product","architecture","baUOXvNBny6TWwuc15FqElb298wYAUTCsj4w4bJOVAQ",{"id":3577,"title":3578,"body":3579,"canonical_url":38,"chapter":38,"date":40,"description":3641,"extension":41,"featured":45,"image":38,"kind":1664,"meta":3642,"navigation":45,"path":3643,"seo":3644,"series":38,"stem":3645,"summary":3646,"tags":3647,"work_slug":38,"__hash__":3650},"writing\u002Fideas\u002F2026-07-14-ludicrous-speed.md","Ludicrous Speed",{"type":7,"value":3580,"toc":3639},[3581,3587,3590,3593,3596,3599,3605,3608,3615,3618,3621,3624,3627,3630,3633,3636],[10,3582,3583,3584,79],{},"We're trying a new way of building products at Commons. I call it ",[515,3585,3586],{},"ludicrous speed",[10,3588,3589],{},"It only works if you suspend disbelief for a while. Let go of a few old assumptions about software development and embrace the vibe.",[10,3591,3592],{},"The reason it works is that almost everything we build shares the same DNA. Forty or so applications. One framework (Nuxt). The same UI library. The same databases. The same deployment pipeline. The same monitoring. Once you've removed enough variation, an LLM stops generating code and starts completing patterns.",[10,3594,3595],{},"Git has become the coordination layer. Ideas become issues. Issues become pull requests. Pull requests become reviews. Reviews become merges. Merges deploy automatically to production. It's a bit like Pieter Levels' philosophy of living in production: ship first, fix quickly, keep moving.",[10,3597,3598],{},"My job has changed more than I expected. I spend almost no time writing code.",[10,3600,3601,3602,79],{},"Instead, I collect feedback and think carefully about specifications. During meetings I'll describe bugs, annotate screenshots, sketch features, and let Claude turn everything into issues. At other times I think about functionality, scope, and (more importantly) what can be removed, simplified, or consolidated. I have LLMs challenge those ideas before turning them into issues. I'll polish them, group a few into epics if it makes sense, and mark them ",[515,3603,3604],{},"Ready",[10,3606,3607],{},"From there, the system takes over.",[10,3609,3610,3611,3614],{},"Cloud instances pick up the work, implement it, review one another, fix what needs fixing, merge to ",[129,3612,3613],{},"main",", and deploy. I mostly watch the conveyor belt, stepping in when the pipeline gets stuck or an agent needs a nudge. Most of the time I simply receive a WhatsApp notification telling me a feature is live.",[10,3616,3617],{},"The remarkable part isn't the automation.",[10,3619,3620],{},"It's the latency.",[10,3622,3623],{},"I can walk out of a product meeting, spend ten minutes polishing the issues, and by the time I've grabbed a coffee, people are using the changes in production. The same applies to ideas from the rest of the team. They create issues, or simply tell someone on the tech team what they want, and minutes later it's real software.",[10,3625,3626],{},"That's what ludicrous speed feels like.",[10,3628,3629],{},"There are no natural pauses anymore. The default state is shipping.",[10,3631,3632],{},"It's not for everyone. Sometimes we deliberately slow things down, add approvals, or gate releases. But for many of our internal products, we've found that continuous deployment with AI in the loop is not only viable, it's genuinely more enjoyable.",[10,3634,3635],{},"I don't know if this is where software development is heading.",[10,3637,3638],{},"I just know I've never had more fun building products.",{"title":35,"searchDepth":36,"depth":36,"links":3640},[],"We're trying a new way of building products at Commons. I call it ludicrous speed.",{},"\u002Fideas\u002F2026-07-14-ludicrous-speed",{"title":3578,"description":3641},"ideas\u002F2026-07-14-ludicrous-speed","How we build products at Commons now. Ideas become issues, agents turn them into merged, deployed software, and the default state is shipping.",[53,54,3648,3649],"claude-code","automation","JcqZqHIfUMjXht1X0Q3gc5FjnPLiRvHl1_rSw52GUX8",{"id":3652,"title":3653,"body":3654,"canonical_url":38,"chapter":38,"date":3757,"description":3658,"extension":41,"featured":45,"image":38,"kind":1664,"meta":3758,"navigation":45,"path":3759,"seo":3760,"series":38,"stem":3761,"summary":3762,"tags":3763,"work_slug":38,"__hash__":3766},"writing\u002Fideas\u002F2026-07-15-the-flame.md","The Flame",{"type":7,"value":3655,"toc":3755},[3656,3659,3662,3665,3668,3671,3674,3677,3680,3683,3686,3689,3692,3695,3698,3701,3704,3707,3710,3713,3716,3719,3722,3725,3728,3731,3734,3737,3740,3743,3746,3749,3752],[10,3657,3658],{},"Life may not be unique.",[10,3660,3661],{},"Given enough planets and enough time, perhaps it emerges over and over again throughout the universe.",[10,3663,3664],{},"I hope that's true.",[10,3666,3667],{},"But hoping new life will appear is a bit like hoping a new flame will suddenly ignite in a cold room.",[10,3669,3670],{},"Maybe it can happen.",[10,3672,3673],{},"If you're trying to keep a fire alive, though, that's not a strategy.",[10,3675,3676],{},"You protect the flames that are already burning.",[10,3678,3679],{},"As far as we know, every living thing on Earth descends from a single chain of life that has remained unbroken for nearly four billion years.",[10,3681,3682],{},"Every tree, every whale, every bacterium, every human being is part of that same continuous flame.",[10,3684,3685],{},"Perhaps that's how we should think about life.",[10,3687,3688],{},"Not as something we own.",[10,3690,3691],{},"Not even as something particularly rare.",[10,3693,3694],{},"But as something extraordinarily precious because, for now, it's the only flame we can actually see.",[10,3696,3697],{},"I sometimes wonder if we've chosen the wrong objective function for civilization.",[10,3699,3700],{},"We optimize for GDP.",[10,3702,3703],{},"For efficiency.",[10,3705,3706],{},"For quarterly earnings.",[10,3708,3709],{},"For convenience.",[10,3711,3712],{},"Even sustainability feels strangely defensive. The goal seems to be simply not to make things worse.",[10,3714,3715],{},"What if the objective were different?",[10,3717,3718],{},"What if our goal was to maximize the flourishing of life?",[10,3720,3721],{},"Not just human life.",[10,3723,3724],{},"Life.",[10,3726,3727],{},"That single change in perspective quietly rearranges everything.",[10,3729,3730],{},"A forest isn't valuable because it stores carbon.",[10,3732,3733],{},"It is valuable because it is life creating the conditions for more life.",[10,3735,3736],{},"Education isn't valuable because it increases salaries.",[10,3738,3739],{},"It is valuable because knowledge compounds across generations.",[10,3741,3742],{},"Scientific research, public health, resilient institutions, open-source software, renewable energy, space exploration: they stop looking like unrelated causes.",[10,3744,3745],{},"They're all investments in the same thing.",[10,3747,3748],{},"Keeping the flame alive.",[10,3750,3751],{},"Perhaps that's all civilization really is.",[10,3753,3754],{},"One generation tending a fire that began billions of years before it arrived, hoping it burns a little brighter after it's gone.",{"title":35,"searchDepth":36,"depth":36,"links":3756},[],"2026-07-15",{},"\u002Fideas\u002F2026-07-15-the-flame",{"title":3653,"description":3658},"ideas\u002F2026-07-15-the-flame","Every living thing on Earth is part of one unbroken chain nearly four billion years old. What if civilization's objective function were keeping that flame alive?",[1668,3764,3765],"life","civilization","lhiPMsA-26cwhhkNcQAjh0LwGkEQIV5JxFdvaWFyn3s",{"id":3768,"title":3769,"body":3770,"canonical_url":38,"chapter":974,"date":3797,"description":3776,"extension":41,"featured":42,"image":38,"kind":43,"meta":3798,"navigation":45,"path":3799,"seo":3800,"series":48,"stem":3801,"summary":3802,"tags":3803,"work_slug":52,"__hash__":3806},"writing\u002Fideas\u002F2026-07-29-anton-13-twelve-days-silence.md","Anton, chapter 13: Twelve days of silence",{"type":7,"value":3771,"toc":3795},[3772,3777,3780,3783,3786,3789,3792],[10,3773,3774],{},[13,3775,3776],{},"2026-07-17 → 2026-08-02",[10,3778,3779],{},"I ask the question almost idly, the way you check a smoke detector you're sure is fine: how are we doing on family-chat monitoring? The answer stops me. Zero messages in seven days. Not a quiet week, zero, and quiet weeks don't read as exactly nothing. The bridge that carries every message between Anton and the family chat has been dead since the seventeenth, rejected with a logout the system took at face value, and for twelve days not one thing noticed. All inbound stopped: no family messages, so watch mode had nothing to watch and memory had nothing to store. All outbound stopped too: every notification failing on a dead socket, every proactive nudge and deploy notice and status question lost without a trace. Two weeks of a system I thought I was watching, watching nothing.",[10,3781,3782],{},"The reason nothing caught it is the part that actually scares me. The health check was green the entire time. It hits the HTTP port, and the HTTP port stays up perfectly well while the underlying socket to the chat network is dead in the water. So the check reported healthy, zero restarts, the picture of a service doing its job, while the service did nothing at all. A health check that measures the wrong thing is worse than no health check, because no health check at least tells you honestly that you don't know. This one manufactured confidence it hadn't earned, and I believed it, because that's what green is for.",[10,3784,3785],{},"Then the second turn of the screw, the one that makes it a trap instead of a bug. Every alert path I have delivers over the same chat channel that's down. The system's way of telling me anything was broken ran through the exact thing that was broken. An outage on that channel is structurally unable to report itself; the alarm and the fire are wired to the same dead wire. I'd built a monitoring story with a single point of failure at the one place a single point of failure is unforgivable, the place where it tells you whether it's alive.",[10,3787,3788],{},"Recovery is fiddly and a little absurd. I reset the auth and re-pair by scanning a code that rotates every twenty seconds, relaying it through a live session fast enough to beat the clock, and the connection comes back. The one thing that goes right on its own is that watch mode resumes without vomiting twelve days of backlog into my phone, because the dedup holds and it knows which messages it has already seen. Small mercy, correctly built earlier, exactly the kind of plumbing that only gets noticed when it would have hurt and didn't.",[10,3790,3791],{},"And while I'm in there, the twin discovery, because these things travel in pairs. The home server's deploy credential had also quietly expired, somewhere in the same stretch. Deploys had been reporting success while the box ran stale code, green checkmarks over a machine that wasn't taking the updates. Two separate silent expirations, same failure signature: a thing that looks fine because the layer reporting on it can't see the layer that died. I fix that one durably, with a credential that doesn't belong to the class of things that expires, so the whole category of \"it quietly lapsed\" stops being possible rather than just being fixed for now.",[10,3793,3794],{},"The lasting fixes are unglamorous and exactly right: a health check that reflects the socket instead of the port, and a re-pair page that turns a silent logout into something I can act on. But the thing I carry out of this fortnight isn't a patch, it's a rule I now believe in my spine. False-green is more dangerous than red, because red gets answered and green gets trusted. And a monitor that can only speak through the channel it's monitoring isn't a monitor, it's a coin flip that lands heads until the day it matters. Everything critical gets a second way to shout now, on a wire that doesn't share fate with the thing it's watching.",{"title":35,"searchDepth":36,"depth":36,"links":3796},[],"2026-07-29",{},"\u002Fideas\u002F2026-07-29-anton-13-twelve-days-silence",{"title":3769,"description":3776},"ideas\u002F2026-07-29-anton-13-twelve-days-silence","The WhatsApp bridge died for twelve days without anyone noticing, because the health check measured the wrong thing and every alert routed through the dead channel.",[52,53,54,3804,3805],"monitoring","reliability","RVZ1oNzqn9cy-xRXQ5NA4sgBT5DQP9ch_m3Zfg48W6k",{"id":3808,"title":3809,"body":3810,"canonical_url":38,"chapter":987,"date":3840,"description":3816,"extension":41,"featured":42,"image":38,"kind":43,"meta":3841,"navigation":45,"path":3842,"seo":3843,"series":48,"stem":3844,"summary":3845,"tags":3846,"work_slug":52,"__hash__":3848},"writing\u002Fideas\u002F2026-08-04-anton-14-guardrail-lied.md","Anton, chapter 14: The guardrail that lied",{"type":7,"value":3811,"toc":3838},[3812,3817,3820,3823,3826,3829,3832,3835],[10,3813,3814],{},[13,3815,3816],{},"2026-08-01 → 2026-08-05",[10,3818,3819],{},"The first unprompted direct message Anton ever sent me arrives without my asking for anything.",[10,3821,3822],{},"Scheduled posts to the group are old news (movie night has run since March), but this is different: nobody scheduled it. Someone in the family group mentions a pickup, and a moment later my phone lights up with a quiet heads-up about it: no question posed, no command given, just Anton deciding on its own that this was worth a nudge. Watch mode is live. It's capped at a couple of interventions a day and silent by default, deliberately timid, because the failure mode of a proactive assistant is being a pest and I'd rather it miss than nag. Seeing it work is the good part of this week. Then I do the thing that always costs me: I audit a single real interaction, Clara's, end to end, and one exchange hands me three bugs.",[10,3824,3825],{},"The first is embarrassing in its simplicity. Every timed event Anton has ever put on the family calendar is two hours off. The model writes a time as plain wall-clock digits, the way a person would, and somewhere down the stack those naive digits get handed to Google as if they were already UTC, while the family very much lives in Paris. So every appointment quietly slid two hours late, for as long as the feature has existed, and nobody's calendar screamed because a wrong time still looks like a time. The fix is to resolve the wall-clock in the family's zone before it ever leaves the box, which is obvious the instant you see it and invisible until you check a real event against a real clock.",[10,3827,3828],{},"The second is quieter and stranger. When Anton sends a message through the proactive path, that message never gets written into the conversation history. So minutes after delivering something to me, asked about it, Anton denies knowing anything about it, and it isn't lying exactly: from where it's standing, that message never happened, because the record only kept the things it received and not the things it said on its own initiative. An assistant that can't remember what it told you is barely less broken than one that can't remember what you told it. The record has to capture what was delivered, not just what was requested.",[10,3830,3831],{},"The third is the one I keep turning over, because it's the smartest part of the system betraying its own purpose. There's a guardrail I built after an earlier incident where Anton announced the time to the whole family with total confidence and total invention, \"it is 15:42\" grounded in nothing. The rule that came out of that: a message about to be broadcast to five people must be backed by evidence, or it gets suppressed and replaced with an honest \"I couldn't find that.\" Good rule. Except this time the gate looks at a turn where Anton has just successfully created Clara's event, decides the confirmation isn't grounded, suppresses the true \"done, I added it,\" and sends Clara \"sorry, I couldn't find that\" instead. The event exists. The action succeeded. And the guardrail against lying produced the lie.",[10,3833,3834],{},"The reason cuts to something real about how much authority you hand a checker. The gate could see the names of the tools the turn had called, but not their results. It knew a calendar tool had been invoked; it could not see that the invocation returned a created event. Blind to the result, it did the conservative thing and assumed nothing happened, and conservative was exactly wrong, because something had. Worse, the success message it suppressed still got written into history, so the audit trail recorded a confirmation Clara never received, which means the very record I'd use to check the system agreed with the system and against reality. A guardrail needs evidence proportional to the authority it holds, and this one held the authority to call Anton a liar while seeing less than the action it was judging.",[10,3836,3837],{},"So the gate learns to read results, not just names: a completed action grounds on the tool result that proves it completed, and a verified write that the in-loop auditor already blessed stops getting second-guessed at the door. The history records what actually went out. The through-line of all three is the same and it's the thing I take into August: the safeguards are only as honest as what they can see, and a check that judges an action while blind to its outcome will, given enough turns, eventually suppress the truth and file a clean report about it. I trust a guardrail now exactly as far as I can see what it saw, and not one step further.",{"title":35,"searchDepth":36,"depth":36,"links":3839},[],"2026-08-04",{},"\u002Fideas\u002F2026-08-04-anton-14-guardrail-lied",{"title":3809,"description":3816},"ideas\u002F2026-08-04-anton-14-guardrail-lied","Watch mode goes live, and three bugs reveal themselves through one real interaction: calendar times two hours off, messages missing from history, and a guardrail that suppressed the truth while blind to the evidence.",[52,53,54,56,3847],"validation","_KonC0LtmQPIRQibUXFFrIgzhGAWrSbWAQPwGUzGH34",{"id":3850,"title":3851,"body":3852,"canonical_url":38,"chapter":38,"date":3840,"description":3856,"extension":41,"featured":42,"image":38,"kind":1664,"meta":4182,"navigation":45,"path":4183,"seo":4184,"series":38,"stem":4185,"summary":4186,"tags":4187,"work_slug":38,"__hash__":4190},"writing\u002Fideas\u002F2026-08-04-how-low-can-inference-go.md","How Low Can Inference Go?",{"type":7,"value":3853,"toc":4176},[3854,3857,3860,3863,3866,3869,3872,3875,3878,3881,3884,3887,3890,3893,3896,3899,3902,3905,3909,3912,4149,4153,4156,4160],[10,3855,3856],{},"On Lex Fridman's podcast, Kieran Kunhya, a longtime contributor to FFmpeg described sections of video-decoding code written directly in assembly, sometimes producing improvements measured in multiples rather than percentages. The software was written for and optimized for each variation of hardware.",[10,3858,3859],{},"The intuition is simple: bare metal is fastest.",[10,3861,3862],{},"Remove enough layers, specialize closely enough to the hardware, and eventually there is nowhere left for inefficiency to hide.",[10,3864,3865],{},"LLMs are not particularly good at writing assembly today. But that may be temporary. Low-level optimization is an attractive problem for machines: generate many candidates, test them for correctness, benchmark them, and keep only the fastest.",[10,3867,3868],{},"The same idea may apply higher up the stack.",[10,3870,3871],{},"Antirez recently built a bespoke inference engine for DeepSeek V4 Flash. By targeting one model rather than supporting every model and every configuration, he was able to outperform more general inference frameworks in important parts of the workload.",[10,3873,3874],{},"This raises a larger question.",[10,3876,3877],{},"How much of the software stack exists because the problem requires it, and how much exists because humans needed help programming computers?",[10,3879,3880],{},"Languages, frameworks, compilers and build systems are intermediate steps between intent and machine execution. Elon Musk has argued that AI will eventually compress that path, moving directly from intent toward machine code.",[10,3882,3883],{},"Perhaps he is right.",[10,3885,3886],{},"But it does not follow that every abstraction should disappear.",[10,3888,3889],{},"Some abstractions exist primarily for human readability. Others provide useful primitives for reasoning about the problem itself: memory, concurrency, data flow, isolation, interfaces and architecture.",[10,3891,3892],{},"Those may remain useful to machines as well as humans.",[10,3894,3895],{},"The future may therefore not be intent translated directly into one opaque binary. It may be a shorter stack made from different abstractions: fewer layers designed around human syntax, and more layers designed for search, verification and specialization.",[10,3897,3898],{},"Bespoke inference engines are an interesting place to watch this happen.",[10,3900,3901],{},"One model. One machine. One workload.",[10,3903,3904],{},"Then ask how low it can go.",[93,3906,3908],{"id":3907},"addendum-the-numbers","Addendum: the numbers",[10,3910,3911],{},"Single DGX Spark (GB10, 128 GB). Same harness for all models: bench.py for throughput, HumanEval pass@1 (greedy, 4096 max tokens), and a 128-prompt reasoning set (57 MMLU-Pro, 57 GPQA Diamond, 14 AIME).",[712,3913,3914,3932],{},[715,3915,3916],{},[718,3917,3918,3920,3923,3926,3929],{},[721,3919],{},[721,3921,3922],{},"Gemma 4 26B-A4B (FP8+MTP, vLLM)",[721,3924,3925],{},"Laguna S 2.1 118B (NVFP4+DFlash, vLLM)",[721,3927,3928],{},"DeepSeek V4 Flash 284B (IQ3_XXS, llama.cpp)",[721,3930,3931],{},"DeepSeek V4 Flash 284B (Q2 imatrix+DSpark, ds4)",[734,3933,3934,3951,3967,3984,4001,4018,4035,4051,4068,4085,4101,4118,4134],{},[718,3935,3936,3939,3942,3945,3948],{},[739,3937,3938],{},"Weights on disk",[739,3940,3941],{},"27 GB",[739,3943,3944],{},"72 GB",[739,3946,3947],{},"103 GB",[739,3949,3950],{},"87 GB",[718,3952,3953,3956,3959,3962,3965],{},[739,3954,3955],{},"Active params",[739,3957,3958],{},"~4B",[739,3960,3961],{},"~8.5B",[739,3963,3964],{},"~13B",[739,3966,3964],{},[718,3968,3969,3972,3975,3978,3981],{},[739,3970,3971],{},"Decode, single stream",[739,3973,3974],{},"57-92 tok\u002Fs",[739,3976,3977],{},"21-27 prose \u002F ~45 code",[739,3979,3980],{},"14.5-16.4 tok\u002Fs",[739,3982,3983],{},"19-27 tok\u002Fs",[718,3985,3986,3989,3992,3995,3998],{},[739,3987,3988],{},"Prefill",[739,3990,3991],{},"~950 tok\u002Fs",[739,3993,3994],{},"~2,700-2,800 tok\u002Fs",[739,3996,3997],{},"~420-450 tok\u002Fs",[739,3999,4000],{},"~950-1,080 tok\u002Fs",[718,4002,4003,4006,4009,4012,4015],{},[739,4004,4005],{},"TTFT, 10K prompt",[739,4007,4008],{},"~10 s",[739,4010,4011],{},"~3.6 s",[739,4013,4014],{},"~20 s",[739,4016,4017],{},"~9.5 s",[718,4019,4020,4023,4026,4029,4032],{},[739,4021,4022],{},"Tool-call turn, 10K ctx",[739,4024,4025],{},"n\u002Fa",[739,4027,4028],{},"4.6 s",[739,4030,4031],{},"24.6 s",[739,4033,4034],{},"13.0 s",[718,4036,4037,4040,4042,4045,4048],{},[739,4038,4039],{},"Concurrent out, 8x5K",[739,4041,4025],{},[739,4043,4044],{},"36.8 tok\u002Fs agg",[739,4046,4047],{},"13.0 tok\u002Fs agg",[739,4049,4050],{},"21.0 tok\u002Fs agg",[718,4052,4053,4056,4059,4062,4065],{},[739,4054,4055],{},"HumanEval pass@1",[739,4057,4058],{},"97.6%",[739,4060,4061],{},"86.6%",[739,4063,4064],{},"93.3%",[739,4066,4067],{},"90.2%",[718,4069,4070,4073,4076,4079,4082],{},[739,4071,4072],{},"Reasoning overall",[739,4074,4075],{},"74.2%",[739,4077,4078],{},"53.1%",[739,4080,4081],{},"64.1%",[739,4083,4084],{},"67.2%",[718,4086,4087,4090,4093,4096,4099],{},[739,4088,4089],{},"MMLU-Pro",[739,4091,4092],{},"78.9%",[739,4094,4095],{},"73.7%",[739,4097,4098],{},"82.5%",[739,4100,4098],{},[718,4102,4103,4106,4109,4112,4115],{},[739,4104,4105],{},"GPQA Diamond",[739,4107,4108],{},"71.9%",[739,4110,4111],{},"38.6%",[739,4113,4114],{},"52.6%",[739,4116,4117],{},"61.4%",[718,4119,4120,4123,4126,4129,4132],{},[739,4121,4122],{},"AIME",[739,4124,4125],{},"64.3%",[739,4127,4128],{},"28.6%",[739,4130,4131],{},"35.7%",[739,4133,4128],{},[718,4135,4136,4139,4142,4145,4147],{},[739,4137,4138],{},"Vision",[739,4140,4141],{},"yes",[739,4143,4144],{},"no",[739,4146,4144],{},[739,4148,4144],{},[2142,4150,4152],{"id":4151},"why-gemma-4-still-carries-the-bag","Why Gemma 4 still carries the bag",[10,4154,4155],{},"For my day-to-day it isn't close. Gemma wins the quality gates I care about (best HumanEval, best reasoning, and it's the only one that takes images), and it's the fastest at the thing you actually watch: streaming tokens. The bigger models each earn a niche: Laguna's 3x prefill makes it the snappiest on cold, fat-context agent turns, and DeepSeek's 284B brain posts the best MMLU-Pro on the box. But my real workload is warm agent loops with cached prefixes, where turns are decode-dominated (Gemma territory), and a 26B running at full FP8 with speculative decoding simply gives up less than a 284B squeezed to 2-3 bits. Until something beats it on quality without trading away the interactive speed, it keeps the production slot.",[93,4157,4159],{"id":4158},"references","References",[536,4161,4162,4169],{},[539,4163,4164],{},[675,4165,4168],{"href":4166,"rel":4167},"https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=nepKKz-MzFM",[679],"FFmpeg: The Incredible Technology Behind Video on the Internet, Lex Fridman Podcast #496",[539,4170,4171],{},[675,4172,4175],{"href":4173,"rel":4174},"https:\u002F\u002Fgithub.com\u002Fantirez\u002Fds4",[679],"antirez\u002Fds4: a bespoke DeepSeek V4 Flash inference engine for Metal, CUDA and ROCm",{"title":35,"searchDepth":36,"depth":36,"links":4177},[4178,4181],{"id":3907,"depth":36,"text":3908,"children":4179},[4180],{"id":4151,"depth":853,"text":4152},{"id":4158,"depth":36,"text":4159},{},"\u002Fideas\u002F2026-08-04-how-low-can-inference-go",{"title":3851,"description":3856},"ideas\u002F2026-08-04-how-low-can-inference-go","The modern software stack lives several layers above bare metal. How much of it exists only because humans were in the loop?",[54,1133,4188,4189],"inference","performance","EBLuNVznAbL4Y3Ko5doqbCcjuWWz9U7M6RI_qEii5OE",{"id":4192,"title":4193,"body":4194,"canonical_url":38,"chapter":38,"date":3840,"description":4198,"extension":41,"featured":45,"image":38,"kind":1664,"meta":4279,"navigation":45,"path":4280,"seo":4281,"series":38,"stem":4282,"summary":4283,"tags":4284,"work_slug":38,"__hash__":4285},"writing\u002Fideas\u002F2026-08-04-time-is-a-security-boundary.md","Time Is a Security Boundary",{"type":7,"value":4195,"toc":4277},[4196,4199,4202,4205,4208,4211,4214,4217,4220,4223,4226,4229,4232,4235,4238,4241,4244,4247,4250,4253,4256,4259,4262,4265,4268,4271,4274],[10,4197,4198],{},"We recently had a security incident. As much as it is uncomfortable, it is part of the life of a modern organisation.",[10,4200,4201],{},"One of the first questions was whether our own AI systems had created the opening. It was a reasonable question. Rapidly evolving practices, loosely managed API keys, exposed environment variables, vibe coding with LLMs, and dubious agent harnesses all create new opportunities to leak credentials or introduce vulnerabilities.",[10,4203,4204],{},"AI also changes the security landscape more broadly. Models can discover vulnerabilities, automate reconnaissance, and chain exploits together at a speed that was previously difficult to achieve.",[10,4206,4207],{},"We have seen the defensive side of that capability too. We use AI extensively to inspect and harden software, and we have built security harnesses that found critical bugs in code that had been running for decades.",[10,4209,4210],{},"SecOps is becoming faster and more dynamic.",[10,4212,4213],{},"But after answering the original question somewhat defensively, I started wondering how much the fundamentals had really changed.",[10,4215,4216],{},"I think that mostly, they have not.",[10,4218,4219],{},"The same rules still apply: separate concerns, grant minimal permissions, isolate systems, create observable boundaries, and assume that individual components will eventually fail.",[10,4221,4222],{},"One underrated strategy may become even more important in an AI-driven security environment: imposing speed limits.",[10,4224,4225],{},"In other words, using time as a security boundary.",[10,4227,4228],{},"Automated attacks thrive on speed. An AI system can probe a weakness, adjust its approach, escalate privileges, and move laterally before a human operator has understood the first alert.",[10,4230,4231],{},"Its advantage comes partly from chaining many small actions together without interruption. The obvious defensive response is to break the chain.",[10,4233,4234],{},"A new credential does not need immediate access to every environment.",[10,4236,4237],{},"A destructive deployment does not need to execute instantly.",[10,4239,4240],{},"A large transfer does not need to clear the moment it is requested.",[10,4242,4243],{},"A sensitive permission change can remain pending before becoming active.",[10,4245,4246],{},"These deliberate slow points create time for detection, review, and intervention.",[10,4248,4249],{},"They also increase the cost and complexity of automation. The point is not that an automated system is incapable of waiting. It is that delay creates opportunities for the surrounding environment to change. Sessions expire. Credentials rotate. State changes. Assumptions become stale. Defensive systems gain time to inspect, interrupt, or contain the workflow.",[10,4251,4252],{},"A chain that can be completed in seconds may be easy to automate. One that must remain valid and coherent across hours or days is a different problem.",[10,4254,4255],{},"Time is not a security boundary because machines cannot wait. It is a security boundary because delay gives the rest of the system time to react.",[10,4257,4258],{},"This is also a useful way to think about AI guardrails.",[10,4260,4261],{},"Guardrails are often imagined as filters that stop a model from producing certain outputs. In operational systems, they should also include permissions, rate limits, staged execution, sandboxes, approval thresholds, and intentional latency.",[10,4263,4264],{},"Their purpose is not to make failure impossible. It is to stop a fast, local failure from becoming an immediate, systemic one.",[10,4266,4267],{},"AI allows attackers to move faster than before. That makes it even more important to decide where sensible speed limits should exist. The same mechanisms also protect against human error, compromised credentials, and poorly designed automation.",[10,4269,4270],{},"Intentional latency is only useful when something is watching. If a permission change remains pending for an hour, that hour should be used: to inspect the request, compare it with normal behaviour, rotate credentials, or interrupt the workflow.",[10,4272,4273],{},"AI gives attackers greater speed, but it also gives defenders the ability to maintain continuous, agentic vigilance, the digital equivalent of a guard walking the perimeter.",[10,4275,4276],{},"The goal is not to prevent every failure. It is to ensure that a small failure cannot move through the entire system faster than we can see it.",{"title":35,"searchDepth":36,"depth":36,"links":4278},[],{},"\u002Fideas\u002F2026-08-04-time-is-a-security-boundary",{"title":4193,"description":4198},"ideas\u002F2026-08-04-time-is-a-security-boundary","Automated attacks thrive on chaining small actions without interruption. Deliberate slow points break the chain: delay gives the rest of the system time to react.",[1625,54,53,3574],"4KfZSdo7TPcDvnDiUX6oEoGofNAnJbHuPpYNIJU-_ag",{"id":4287,"title":4288,"body":4289,"demo":38,"description":4307,"extension":41,"featured":45,"image":38,"meta":4308,"name":4288,"navigation":45,"path":4309,"seo":4310,"stem":4311,"summary":4312,"tags":4313,"url":4317,"when":4318,"where":4319,"__hash__":4320},"work\u002Fwork\u002Fanton.md","Anton",{"type":7,"value":4290,"toc":4305},[4291,4298],[10,4292,4293,4294,4297],{},"Anton is a personal agent OS, deployed on a DGX Spark on my home network. About ten specialized domain agents (home, media, coder, research, admin, knowledge, syndic, dev, quality) coordinate via typed delegates. Skills run in a sandboxed Deno runtime, hot-reloadable. Layered memory (context, history, facts), policy engine evaluating every action, thread-aware interruption with cancel cascade, nightly self-improvement loop driven by issues, evaluation infrastructure under ",[129,4295,4296],{},"packages\u002Fagent-quality",". Local LLM inference via vLLM behind a LiteLLM gateway over Tailscale.",[10,4299,4300,4301,4304],{},"Anton is a decade-later realization of ",[13,4302,4303],{},"Bots for Humanity",", a personal-agent \u002F digital-twin concept I worked on as my MBA capstone in 2015 and presented to contacts at Facebook. Design philosophy: ingest cheap, process lazy, enrich on demand. Explore agentic, build deterministic.",{"title":35,"searchDepth":36,"depth":36,"links":4306},[],"Anton is a personal agent OS, deployed on a DGX Spark on my home network. About ten specialized domain agents (home, media, coder, research, admin, knowledge, syndic, dev, quality) coordinate via typed delegates. Skills run in a sandboxed Deno runtime, hot-reloadable. Layered memory (context, history, facts), policy engine evaluating every action, thread-aware interruption with cancel cascade, nightly self-improvement loop driven by issues, evaluation infrastructure under packages\u002Fagent-quality. Local LLM inference via vLLM behind a LiteLLM gateway over Tailscale.",{},"\u002Fwork\u002Fanton",{"description":4307},"work\u002Fanton","Family logistics scatter across apps, inboxes, and memory; Anton is a personal agent OS that runs them in production from hardware at home.",[53,4314,4315,54,4316,1134],"typescript","deno","self-hosted","https:\u002F\u002Fgithub.com\u002Flucbocahut\u002Fanton","2024-12-01","Paris","OXLMu-spymcpGsgsicDfI-hs4bFBLFXMEfiM168dYnY",1793869666068]