How Low Can Inference Go?
On Lex Fridman's podcast, Kieran Kunhya, a longtime contributor to FFmpeg described sections of video-decoding code written directly in assembly, sometimes producing improvements measured in multiples rather than percentages. The software was written for and optimized for each variation of hardware.
The intuition is simple: bare metal is fastest.
Remove enough layers, specialize closely enough to the hardware, and eventually there is nowhere left for inefficiency to hide.
LLMs are not particularly good at writing assembly today. But that may be temporary. Low-level optimization is an attractive problem for machines: generate many candidates, test them for correctness, benchmark them, and keep only the fastest.
The same idea may apply higher up the stack.
Antirez recently built a bespoke inference engine for DeepSeek V4 Flash. By targeting one model rather than supporting every model and every configuration, he was able to outperform more general inference frameworks in important parts of the workload.
This raises a larger question.
How much of the software stack exists because the problem requires it, and how much exists because humans needed help programming computers?
Languages, frameworks, compilers and build systems are intermediate steps between intent and machine execution. Elon Musk has argued that AI will eventually compress that path, moving directly from intent toward machine code.
Perhaps he is right.
But it does not follow that every abstraction should disappear.
Some abstractions exist primarily for human readability. Others provide useful primitives for reasoning about the problem itself: memory, concurrency, data flow, isolation, interfaces and architecture.
Those may remain useful to machines as well as humans.
The future may therefore not be intent translated directly into one opaque binary. It may be a shorter stack made from different abstractions: fewer layers designed around human syntax, and more layers designed for search, verification and specialization.
Bespoke inference engines are an interesting place to watch this happen.
One model. One machine. One workload.
Then ask how low it can go.
Addendum: the numbers
Single DGX Spark (GB10, 128 GB). Same harness for all models: bench.py for throughput, HumanEval pass@1 (greedy, 4096 max tokens), and a 128-prompt reasoning set (57 MMLU-Pro, 57 GPQA Diamond, 14 AIME).
| Gemma 4 26B-A4B (FP8+MTP, vLLM) | Laguna S 2.1 118B (NVFP4+DFlash, vLLM) | DeepSeek V4 Flash 284B (IQ3_XXS, llama.cpp) | DeepSeek V4 Flash 284B (Q2 imatrix+DSpark, ds4) | |
|---|---|---|---|---|
| Weights on disk | 27 GB | 72 GB | 103 GB | 87 GB |
| Active params | ~4B | ~8.5B | ~13B | ~13B |
| Decode, single stream | 57-92 tok/s | 21-27 prose / ~45 code | 14.5-16.4 tok/s | 19-27 tok/s |
| Prefill | ~950 tok/s | ~2,700-2,800 tok/s | ~420-450 tok/s | ~950-1,080 tok/s |
| TTFT, 10K prompt | ~10 s | ~3.6 s | ~20 s | ~9.5 s |
| Tool-call turn, 10K ctx | n/a | 4.6 s | 24.6 s | 13.0 s |
| Concurrent out, 8x5K | n/a | 36.8 tok/s agg | 13.0 tok/s agg | 21.0 tok/s agg |
| HumanEval pass@1 | 97.6% | 86.6% | 93.3% | 90.2% |
| Reasoning overall | 74.2% | 53.1% | 64.1% | 67.2% |
| MMLU-Pro | 78.9% | 73.7% | 82.5% | 82.5% |
| GPQA Diamond | 71.9% | 38.6% | 52.6% | 61.4% |
| AIME | 64.3% | 28.6% | 35.7% | 28.6% |
| Vision | yes | no | no | no |
Why Gemma 4 still carries the bag
For my day-to-day it isn't close. Gemma wins the quality gates I care about (best HumanEval, best reasoning, and it's the only one that takes images), and it's the fastest at the thing you actually watch: streaming tokens. The bigger models each earn a niche: Laguna's 3x prefill makes it the snappiest on cold, fat-context agent turns, and DeepSeek's 284B brain posts the best MMLU-Pro on the box. But my real workload is warm agent loops with cached prefixes, where turns are decode-dominated (Gemma territory), and a 26B running at full FP8 with speculative decoding simply gives up less than a 284B squeezed to 2-3 bits. Until something beats it on quality without trading away the interactive speed, it keeps the production slot.