How Continuous Batching Enables High-Concurrency LLM Inference
Naive batching holds every slot until the longest request finishes. Continuous batching swaps requests in and out at every forward step, which means packing prefill and decode tokens into one flat input and splitting attention back out per request with a cu_seqlens boundary table.