Lead ML Engineer at Sprinklr
Sanjay Budhala
I build scalable AI systems. I also climb mountains.
I design and build LLM-powered copilots. Here I write about AI systems, lessons from infrastructure and MLOps, and the outdoors.
Blog
All posts- Batching LLM Requests: Static, Dynamic and Continuous
Static batching at output-length CV of about 1 keeps only 33% of its decode slots useful, and throughput falls from 168 to 61 tok/s. Each fix, from timeouts to continuous batching to chunked prefill, creates the next problem, measured on a GPT-2 engine built from scratch.
- Why LLM Decoding Is Slow, and Why Batching Is (Almost) Free
A 100-token answer from GPT-2 needs 60 ms of math and takes 2 seconds. Following the bottleneck from a faster chip to the roofline, batching, the KV cache, GQA and goodput, measured on an inference engine built from scratch.
- How LLMs Generate Text, and What the KV Cache Actually Fixes
Generation is a sequential loop, the naive version is quadratic, and the KV cache should make it 260x faster. Measured on GPT-2 built from scratch, it makes it 6x faster. Here is why.
- Agent Memory: Why Every Strategy Trades One Failure for Another
Four ways to give an agent memory — buffer, summarization, retrieval, and graph — tested hands-on, with the failure case that breaks each one.
- Building a Parameter-Aware Rate Limiter SDK with Redis, Lua, and MongoDB
A technical deep-dive into designing a distributed, parameter-aware rate limiting SDK that handles multi-tenant workloads with heterogeneous payload contracts.
Projects
All projectsMercari AI Shopper
Five sequential agents turn a natural-language request into Mercari Japan searches, scrape listings with Playwright, and rank them with RAG. English and Japanese.
Research Copilot
A local RAG system with hybrid BM25 and dense retrieval.
LLM inference from scratch
GPT-2 inference built by hand: KV caching, prefill and decode, sampling and batching.