Blog
Long-form posts on LLM systems, agents, and ML infrastructure.
2026
- Batching LLM Requests: Static, Dynamic and Continuous
Static batching at output-length CV of about 1 keeps only 33% of its decode slots useful, and throughput falls from 168 to 61 tok/s. Each fix, from timeouts to continuous batching to chunked prefill, creates the next problem, measured on a GPT-2 engine built from scratch.
- Why LLM Decoding Is Slow, and Why Batching Is (Almost) Free
A 100-token answer from GPT-2 needs 60 ms of math and takes 2 seconds. Following the bottleneck from a faster chip to the roofline, batching, the KV cache, GQA and goodput, measured on an inference engine built from scratch.
- How LLMs Generate Text, and What the KV Cache Actually Fixes
Generation is a sequential loop, the naive version is quadratic, and the KV cache should make it 260x faster. Measured on GPT-2 built from scratch, it makes it 6x faster. Here is why.
- Agent Memory: Why Every Strategy Trades One Failure for Another
Four ways to give an agent memory — buffer, summarization, retrieval, and graph — tested hands-on, with the failure case that breaks each one.
2025
- Building a Parameter-Aware Rate Limiter SDK with Redis, Lua, and MongoDB
A technical deep-dive into designing a distributed, parameter-aware rate limiting SDK that handles multi-tenant workloads with heterogeneous payload contracts.