The Smallest Model I've Ever Shipped

July 16, 2026

A few weeks ago I got a request that sounded simple: take TOBA — an open 1.2-billion-parameter model trained on Indonesian plus regional languages like Batak Toba, Javanese, Sundanese, and Minangkabau — and embed it into the cultural chatbot I work on, next to our existing brain (a frontier model behind an API, backed by RAG).

My first reaction was to quietly list the reasons this wouldn't work.

• 1.2B parameters. Our main pipeline calls a model several hundred times that size.
• A 1,024-token context window. One RAG chunk would eat the whole thing.
• Thin conversational training data. This is a research model, not a chat product.

I shipped it anyway. It's live in production now, and I'm glad I was wrong — but not for the reason I expected. The interesting engineering wasn't inference. It was three things I didn't see coming: the product framing, what a 1,024-token budget does to your architecture, and serving a model that has two caches.

The model card is the UI

Here's the thought experiment that changed my mind. Put a 1.2B model in the same dropdown as a frontier model, and users will ask it the same questions. It will answer confidently, wrongly, and in beautiful grammar — and everyone will conclude it's broken.

Frame the same model as "an experimental research model for Nusantara languages — try translation," and suddenly it's impressive. Because translation between regional languages is genuinely its trained strength, and it's something the big API model does worse. Frontier models don't really speak Batak.

So instead of a dropdown entry, TOBA became a visibly distinct mode: an honest capability card as the empty state (small research model, no knowledge base, answers can be wrong, keep it short), a translation composer as the default tab, and free chat as the secondary, caveated option. Deterministic decoding for translation, temperature for chat.

The evaluation runs convinced me this framing was right. The good is really good:

ID → Batak Toba:  "Selamat pagi, apa kabar?"
              →   "selamat pagi, aha do hamu?"        (258 ms)

ID → Sundanese:   "hatur nuhun pisan kana bantosanana,
                   mugia urang tiasa patepang deui minggu payun"
                   — it picked the polite register on its own

And the failures are exactly the ones you should design around. I asked it — in Javanese — about North Sumatran food. It answered in fluent Javanese, listing… Javanese dishes. Another time, a "supported" translation pair quietly slipped a word from a completely different regional language into the output. Language-matching is trained-in and works. Factual grounding is not there, and no context-stuffing fixes that in 1,024 tokens.

The lesson I keep coming back to: for small models, framing does more than fine-tuning. The UI is the model card.

What a 1,024-token budget forces on you

Everyone writes about million-token context windows. Nobody warned me that a 1,024-token budget is a stricter teacher.

Your character-count heuristics are lies. TOBA uses a syllabic tokenizer (~2.8k vocab) — tokens are syllables, not BPE subwords. Every char-based input limit I guessed in the frontend was wrong in both directions. The fix was admitting the server is the only honest token counter and exposing a tiny /v1/tokens/count endpoint backed by the real tokenizer.

History is a budget line, not a list. Multi-turn chat gets:

budget = ctx_len (1024) − max_new_tokens − safety_margin

<|user|>oldest ──✂── dropped first
<|assistant|>...
<|user|>newest ──── always kept
<|user|>current prompt<|assistant|>

Turns are admitted newest-first until the budget runs out. Translation requests skip history entirely — each one is single-shot, because conversation memory would evict the very text being translated.

Long inputs become a streaming protocol. Translating a document means chunking by paragraph, then sentence, translating each chunk deterministically, and streaming per-chunk progress events (part 2/3...) so you watch the document translate itself. The context limit leaked all the way up into our SSE event vocabulary. I've made peace with that.

And one trap I want on the record: when generation would overflow the window, the engine slides the KV cache — but this model uses learned absolute position embeddings, so the trimmed cache entries keep their original positions while new tokens keep counting up. Nothing crashes. Quality just quietly degrades past the boundary. I've been bitten by position-vs-cache mismatches before — I wrote about debugging exactly that class of KV cache bug — so this time we enforce prompt + output ≤ 1024 upstream and treat the slide as a safety net. Silent degradation is worse than a crash, because nobody files a bug report for "slightly worse."

One thing the small window gives you for free: we skipped prefix caching without regret. At 1,024 tokens, prefill is one cheap forward pass — decode is everything. Constraints kill some optimizations and make others unnecessary.

One model, two caches

TOBA's architecture (per the team's published research) is a GPT-2 backbone with an Engram module — a learned n-gram hash memory with bigram/trigram pathways and gating, designed for the morphology of agglutinative languages. The serving consequence nobody's blog prepared me for: incremental decoding threads two cache states through every step. The regular attention KV cache, and the Engram state riding along beside it like a second cache.

Was the plumbing worth it? I A/B'd it on the real weights with greedy decoding — so cached and uncached produce byte-identical output, and any difference is pure compute path:

                      48 new tokens     throughput
KV + Engram cache ON       4.3 s        11.2 tok/s
cache OFF                 13.8 s         3.5 tok/s
                                         ─────────
                                          3.2× — at only 48 tokens

The uncached path re-runs the full sequence per token, so the gap widens quadratically with length. My first smoke test ran on my MacBook's CPU at ~12 tokens/second — usable, weirdly, for a demo. In production, on a single mid-range GPU, the service sustains 46–52 tokens/second (measured this week against the live endpoint: 150-token generation in 2.9 s, a short translation in 258 ms).

Then the question that actually kept me up: how do you serve a model with no batching? The research inference code is strictly batch-size-one. My ladder, in order of laziness:

1. Admission control as an API contract. A semaphore admits N generations, a bounded queue holds the rest, and overflow returns 503 BUSY with a machine-readable code that the frontend renders as "busy, try again." Crucially, that code is distinct from our "upstream provider down" error — a busy local model must never trip the site-wide maintenance banner. Failure-domain isolation applies to error codes, not just infrastructure.

2. Measure whether concurrency > 1 helps at all. Sequential one-token-at-a-time decode doesn't saturate the hardware, so a second concurrent stream fills the idle units. Measured: +43% aggregate throughput on CPU, +23% on GPU at concurrency 2, with each request ~1.5× slower. Worth it — the second user waits four seconds instead of five. Past that, latency degrades faster than throughput grows.

3. Processes, not threads. No custom batching. Weights are ~2.5 GB in bf16, per-stream cache is ~0.2 GB — whole extra processes are affordable. Writing a continuous-batching scheduler around a two-cache custom architecture is weeks of subtle bugs for an experimental feature. Boring won.

One supply-chain habit that earned its keep: the model's code and weights co-evolve in one public repository, so the service pins a single revision and loads both from the same snapshot — never main. We caught a research checkout and the published weights drifting to different layer counts. Pinning turns that class of surprise into a version bump you make on purpose.

What I'd tell you to steal

1. For small models, pave the path to what the model is actually good at. The UI is the model card.
2. A 1,024-token budget is a forcing function — real token counting, budgeted history, chunked streaming — and it makes some optimizations free to skip.
3. Benchmark your caches on your model. Our 3.2× from correct cache plumbing beat any hardware upgrade we could have bought.
4. For batchless models: admission control is an API contract, measured concurrency beats assumed concurrency, and processes beat clever.
5. Pin code and weights as one artifact.

A 1.2B model will never out-think a frontier API. But watching "aha do hamu?" stream back syllable by syllable — in a language most language technology has never bothered to learn — is the most satisfying thing I've shipped this year.