kv-cache

Static batching at output-length CV of about 1 keeps only 33% of its decode slots useful, and throughput falls from 168 to 61 tok/s. Each fix, from timeouts to continuous batching to chunked prefill, creates the next problem, measured on a GPT-2 engine built from scratch.

• llm-inference
• batching
• kv-cache
• performance

A 100-token answer from GPT-2 needs 60 ms of math and takes 2 seconds. Following the bottleneck from a faster chip to the roofline, batching, the KV cache, GQA and goodput, measured on an inference engine built from scratch.

• llm-inference
• kv-cache
• batching
• performance

Generation is a sequential loop, the naive version is quadratic, and the KV cache should make it 260x faster. Measured on GPT-2 built from scratch, it makes it 6x faster. Here is why.

• llm-inference
• kv-cache
• performance