Anton, chapter 10: Making Gemma fly

2026-06-06 → 2026-06-07

The local model on the Spark is Nemotron when this starts: a reasoning model from one vendor, around fifty-eight tokens a second, held together by two unmerged patches I'd backported by hand so the image would even load it. It's the third occupant of the slot since spring (the Gemma from chapter 7 gave way to a Qwen release, which gave way to this), each swap chasing the next promising checkpoint. Nemotron ships no drafter and no speculative decoding; the whole bet was that distillation-time accuracy would carry it. With a fresh wave of Gemma 4 releases out, multimodal and ours to shape, I move the local model back to it, and it settles into the high forties. That should read as progress. Instead it nags at the line I left open in the spring: speculative decoding, a 1.5x I never cashed in. The question sharpens: is the high forties the ceiling of this hardware, or just the ceiling of what I bothered to try? I decide to find out properly, with numbers, and not stop at the first comfortable answer.

The obvious lever is a better quantization, and there's a fresh one to chase: a quantization-aware-training release, weights trained to survive 4-bit. The theory is seductive, full quality at a quarter of the memory. I fan out a small army of research agents and they come back with the same verdict from five angles: it does nothing for us. The Spark is memory-bandwidth-bound, and any 4-bit format moves the same bytes per token; the number that decides everything is bandwidth, and no checkpoint changes it. Worse, the format the box already runs gets its storage win but not its compute win, because this silicon has no native 4-bit math. A second, identical box has joined the network by now, and benchmarking the same checkpoint on both hands me the sharpest version of the point: one serving image does 52 tokens a second where another does 30, on the same hardware, because only one of them ships the right kernels. The image, not the silicon, sets the floor. I'm carrying the cost of the format (a patched model file, a software-emulated kernel, an image that loads it only if you hold your mouth right) and collecting none of the payoff. Quantization was never the lever.

So I look at the machines I already own. My work Mac and an idle laptop both have more memory bandwidth than the Spark, and decode is bandwidth-bound, so on paper they should win. They lose, badly: the same model runs slower on twice the bandwidth, because Apple's mixture-of-experts kernels pull a third of what the datacenter stack does. Bandwidth sets the ceiling; the kernel decides where you land under it. That detour cost me a clean hypothesis and bought a real rule: the spec sheet is not the benchmark.

The lever turns out to be the one I'd left open. Multi-token prediction: a tiny drafter guesses the next few tokens, the big model verifies them in a single pass, and on a bandwidth-bound box that is the only trick that beats the wall, because you pay for one weight read and get several tokens back. A community write-up claims triple the baseline on this exact hardware, and I want it. I match the recipe to the letter and land at half the headline. The gap is not a missing flag; I run their command verbatim and my baseline matches theirs to the decimal. The gap is what "tokens per second" even means. Their number is a harness measuring pure decode with the prefill stripped away; mine is what a caller actually waits for, wall clock, everything included. Both are true. They measure different things. The honest figure, the one a person feels, is a clean 1.4x to 1.9x depending on how predictable the text is, and I write that one down instead of the prettier one.

Then I test it the way Anton is really used: the actual router prompt, a real conversation, sixteen of them at once. The number that matters isn't single-stream at all. When someone is mid-conversation the whole prefix is already cached, and the answer comes back in half a second instead of five, ten times faster on the most common path there is. The win was never a single thing. It was two orthogonal levers, speculation and caching, neither of which has anything to do with the quantization I'd spent days circling.

Last, I try to break it. Five hundred requests at once, and it just queues, never dropping one, the chip barely warm; the old warning about thermal shutdowns at high concurrency refuses to reproduce under a burst, though I don't kid myself that a few minutes of burst is the same as hours of sustained batch on a box that lives in a warm room. So this is where it lands: the local model answers a live conversation in half a second, holds north of a thousand tokens a second across a crowd, and fails over to a second box if the first ever stumbles. The question I started with has an answer. The ceiling I'd accepted was the ceiling of one idea, and most of the climb past it was just measuring honestly enough to see where the floor actually was.