Continuous-Batching on Chi's Bloghttps://zhangchitc.github.io/tags/continuous-batching/Recent content in Continuous-Batching on Chi's BlogHugoen-usFri, 09 Oct 2026 16:23:00 +0800How Continuous Batching Enables High-Concurrency LLM Inferencehttps://zhangchitc.github.io/posts/continuous-batching/Fri, 09 Oct 2026 16:23:00 +0800https://zhangchitc.github.io/posts/continuous-batching/Naive batching holds every slot until the longest request finishes. Continuous batching swaps requests in and out at every forward step, which means packing prefill and decode tokens into one flat input and splitting attention back out per request with a cu_seqlens boundary table.