Sanjay BudhalaNotes on building LLM systems, ML infrastructure and the occasional mountain.https://sanjaybudhala.com/Batching LLM Requests: Static, Dynamic and Continuoushttps://sanjaybudhala.com/blog/batching-llm-requests-static-dynamic-and-continuous/https://sanjaybudhala.com/blog/batching-llm-requests-static-dynamic-and-continuous/Static batching at output-length CV of about 1 keeps only 33% of its decode slots useful, and throughput falls from 168 to 61 tok/s. Each fix, from timeouts to continuous batching to chunked prefill, creates the next problem, measured on a GPT-2 engine built from scratch.Wed, 21 Oct 2026 00:00:00 GMTllm-inferencebatchingkv-cacheperformanceWhy LLM Decoding Is Slow, and Why Batching Is (Almost) Freehttps://sanjaybudhala.com/blog/why-llm-decoding-is-slow-and-why-batching-is-almost-free/https://sanjaybudhala.com/blog/why-llm-decoding-is-slow-and-why-batching-is-almost-free/A 100-token answer from GPT-2 needs 60 ms of math and takes 2 seconds. Following the bottleneck from a faster chip to the roofline, batching, the KV cache, GQA and goodput, measured on an inference engine built from scratch.Mon, 19 Oct 2026 00:00:00 GMTllm-inferencekv-cachebatchingperformanceHow LLMs Generate Text, and What the KV Cache Actually Fixeshttps://sanjaybudhala.com/blog/llm-inference-math-problem-memory-problem/https://sanjaybudhala.com/blog/llm-inference-math-problem-memory-problem/Generation is a sequential loop, the naive version is quadratic, and the KV cache should make it 260x faster. Measured on GPT-2 built from scratch, it makes it 6x faster. Here is why.Mon, 19 Oct 2026 00:00:00 GMTllm-inferencekv-cacheperformanceAgent Memory: Why Every Strategy Trades One Failure for Anotherhttps://sanjaybudhala.com/blog/agent-memory-why-every-strategy-trades-one-failure-for-another/https://sanjaybudhala.com/blog/agent-memory-why-every-strategy-trades-one-failure-for-another/Four ways to give an agent memory — buffer, summarization, retrieval, and graph — tested hands-on, with the failure case that breaks each one.Thu, 30 Jul 2026 00:00:00 GMTagentic-aiagent-memoryBuilding a Parameter-Aware Rate Limiter SDK with Redis, Lua, and MongoDBhttps://sanjaybudhala.com/blog/building-a-parameter-aware-rate-limiter-sdk-with-redis-lua-and-mongodb/https://sanjaybudhala.com/blog/building-a-parameter-aware-rate-limiter-sdk-with-redis-lua-and-mongodb/A technical deep-dive into designing a distributed, parameter-aware rate limiting SDK that handles multi-tenant workloads with heterogeneous payload contracts.Wed, 15 Oct 2025 00:00:00 GMTsystem-designinfrastructuredistributed-systemsrate-limitingmicroservices