Why FlashAttention is So Fast
FlashAttention does every multiplication standard attention does, yet runs several times faster. We build it up step by step from the GPU memory hierarchy: kernel fusion, safe softmax, online softmax, and tiling.
Attention
How Much GPU Memory Does the KV Cache Really Take?
We build the KV cache size formula one factor at a time, from single-head attention to multi-head, stacked layers and MQA/GQA, then check it against a real calculator: 13.73 GiB for 100,000 tokens on Qwen3-8B.
KV Cache vs. Prompt Cache: What's the Difference?
A prompt cache doesn’t store your last answer. We walk through prefill, decode and attention to see what the KV cache keeps, and how a prompt cache reuses it across requests.
Why Transformers Need Positional Encoding
Attention can’t tell “dog bites man” from “man bites dog”. We try integer, fractional and binary encodings, see where each one breaks, and arrive at sines and cosines: a set of clock hands spinning at different speeds.