The Smallest Model I've Ever Shipped
July 16, 2026
A few weeks ago I got a request that sounded simple: take TOBA (an open 1.2-billion-parameter model trained on Indonesian plus regional languages like Batak Toba, Javanese, Sundanese, and Minangkabau) and embed it into the cultural chatbot I work on, next to our existing brain (a frontier model behind an API, backed by RAG).
My first reaction was to list the reasons this wouldn't work.
- 1.2B parameters. Our main pipeline calls a model several hundred times that size.
- A 1,024-token context window. One RAG chunk would eat the whole thing.
- Thin conversational training data. This is a research model, not a chat product.
I shipped it anyway, and it's live in production. The hard parts weren't the ones on my list. They were the product framing, what a 1,024-token budget does to your architecture, and serving a model that has two caches.
The model card is the UI
Put a 1.2B model in the same dropdown as a frontier model and users will ask it the same questions. It will answer confidently, wrongly, and in beautiful grammar, and everyone will conclude it's broken.
Frame the same model as "an experimental research model for Nusantara languages: try translation," and suddenly it's impressive, because translation between regional languages is what it was trained for and the big API model does it worse. Frontier models don't really speak Batak.
So instead of a dropdown entry, TOBA became a visibly distinct mode: an honest capability card as the empty state (small research model, no knowledge base, answers can be wrong, keep it short), a translation composer as the default tab, and free chat as the secondary, caveated option. Deterministic decoding for translation, temperature for chat.
The evaluation runs backed that framing up. The good cases:
ID → Batak Toba: "Selamat pagi, apa kabar?"
→ "selamat pagi, aha do hamu?" (258 ms)
ID → Sundanese: "hatur nuhun pisan kana bantosanana,
mugia urang tiasa patepang deui minggu payun"
— it picked the polite register on its ownThe failures are the kind you design around. I asked it about North Sumatran food, in Javanese. It answered in fluent Javanese, listing… Javanese dishes. Another time, a "supported" translation pair slipped a word from a completely different regional language into the output. Language-matching is trained-in and works. Factual grounding is not there, and no context-stuffing fixes that in 1,024 tokens.
For small models, framing does more than fine-tuning.
What a 1,024-token budget forces on you
Everyone writes about million-token context windows. Almost nobody writes about what a 1,024-token one does to a design.
Your character-count heuristics are lies. TOBA uses a syllabic tokenizer (~2.8k vocab), so tokens are syllables, not BPE subwords. Every char-based input limit I guessed in the frontend was wrong in both directions. The fix was to let the server count, through a tiny /v1/tokens/count endpoint backed by the real tokenizer.
History is a budget line, not a list. Multi-turn chat gets:
budget = ctx_len (1024) − max_new_tokens − safety_margin
<|user|>oldest ──✂── dropped first
<|assistant|>...
<|user|>newest ──── always kept
<|user|>current prompt<|assistant|>Turns are admitted newest-first until the budget runs out. Translation requests skip history entirely, because each one is single-shot and conversation memory would evict the very text being translated.
Long inputs become a streaming protocol. Translating a document means chunking by paragraph, then sentence, translating each chunk deterministically, and streaming per-chunk progress events (part 2/3...) so you watch the document translate itself. The context limit leaked all the way up into our SSE event vocabulary. I've made peace with that.
When generation would overflow the window, the engine slides the KV cache. This model uses learned absolute position embeddings, so the trimmed cache entries keep their original positions while new tokens keep counting up. Nothing crashes; quality degrades past the boundary. I've been bitten by position-vs-cache mismatches before (I wrote about debugging exactly that class of KV cache bug), so this time we enforce prompt + output ≤ 1024 upstream and treat the slide as a safety net. Silent degradation is worse than a crash, because nobody files a bug report for "slightly worse."
The small window also let us skip prefix caching without regret. At 1,024 tokens, prefill is one cheap forward pass and decode is everything.
One model, two caches
TOBA's architecture (per the team's published research) is a GPT-2 backbone with an Engram module, a learned n-gram hash memory with bigram/trigram pathways and gating, designed for the morphology of agglutinative languages. For serving, that means incremental decoding threads two cache states through every step: the regular attention KV cache, and the Engram state riding along beside it.
To see if the plumbing was worth it, I A/B'd it on the real weights with greedy decoding, so cached and uncached produce byte-identical output and any difference is pure compute path:
48 new tokens throughput
KV + Engram cache ON 4.3 s 11.2 tok/s
cache OFF 13.8 s 3.5 tok/s
─────────
3.2× — at only 48 tokensThe uncached path re-runs the full sequence per token, so the gap widens quadratically with length. My first smoke test ran on my MacBook's CPU at ~12 tokens/second, which was usable, weirdly, for a demo. In production, on a single mid-range GPU, the service sustains 46–52 tokens/second (measured this week against the live endpoint: 150-token generation in 2.9 s, a short translation in 258 ms).
The research inference code is strictly batch-size-one, so the next question was how to serve it with no batching. My ladder, in order of laziness:
Admission control as an API contract. A semaphore admits N generations, a bounded queue holds the rest, and overflow returns
503 BUSYwith a machine-readable code that the frontend renders as "busy, try again." That code is distinct from our "upstream provider down" error, because a busy local model must never trip the site-wide maintenance banner.Measure whether concurrency > 1 helps at all. Sequential one-token-at-a-time decode doesn't saturate the hardware, so a second concurrent stream fills the idle units. Measured: +43% aggregate throughput on CPU and +23% on GPU at concurrency 2, with each request ~1.5× slower. Worth it: the second user waits four seconds instead of five. Past that, latency degrades faster than throughput grows.
Processes, not threads. No custom batching. Weights are ~2.5 GB in bf16, per-stream cache is ~0.2 GB, so whole extra processes are affordable. Writing a continuous-batching scheduler around a two-cache custom architecture is weeks of subtle bugs for an experimental feature.
The model's code and weights co-evolve in one public repository, so the service pins a single revision and loads both from the same snapshot, never main. We caught a research checkout and the published weights disagreeing on layer count. Pinning makes that kind of drift a version bump you make on purpose.
A 1.2B model won't out-think a frontier API. Watching "aha do hamu?" stream back syllable by syllable, in a language most language technology skips, is the most satisfying thing I've shipped this year.