- Where
- Bentara Fraktal Indonesia Technologies
- When
- 2025–2026
- Role
- AI Full Stack Engineer
- Stack
- Toba language models, Custom KV cache, Concurrency testing
Making small language models keep up under load
Toba is a family of in-house trilingual language models. I engineered a custom KV cache layer for their inference path, then validated it under heavy concurrent load to make sure the gains held.
Outcome Throughput held up under concurrent load testing.