Notes on how LLMs and coding agents work under the hood.
How Mixture of Experts Scales Model Parameters
Mixture of Experts lets a model’s parameter count grow tens of times over while the compute per token barely moves. We build it up step by step: copying the FFN into experts, the router, dispatch and combine, top-K routing, and the auxiliary loss that keeps experts balanced.
Why FlashAttention is So Fast
FlashAttention does every multiplication standard attention does, yet runs several times faster. We build it up step by step from the GPU memory hierarchy: kernel fusion, safe softmax, online softmax, and tiling.
How PagedAttention Works: Paging the KV Cache Like an Operating System
Why serving LLMs fragments GPU memory, and how vLLM’s PagedAttention fixes it with OS-style paging: physical blocks, a block table, chained block hashes, and a walkthrough of how blocks are allocated, shared, freed and evicted.
How Much GPU Memory Does the KV Cache Really Take?
We build the KV cache size formula one factor at a time, from single-head attention to multi-head, stacked layers and MQA/GQA, then check it against a real calculator: 13.73 GiB for 100,000 tokens on Qwen3-8B.
KV Cache vs. Prompt Cache: What's the Difference?
A prompt cache doesn’t store your last answer. We walk through prefill, decode and attention to see what the KV cache keeps, and how a prompt cache reuses it across requests.
Why Transformers Need Positional Encoding
Attention can’t tell “dog bites man” from “man bites dog”. We try integer, fractional and binary encodings, see where each one breaks, and arrive at sines and cosines: a set of clock hands spinning at different speeds.