https://zhangchitc.github.io/tags/rope/Recent content in Rope on Chi's BlogHugoen-usTue, 27 Oct 2026 01:44:00 +0800https://zhangchitc.github.io/posts/multi-head-latent-attention/Tue, 27 Oct 2026 01:44:00 +0800https://zhangchitc.github.io/posts/multi-head-latent-attention/MLA caches one 512-dimensional latent per token instead of 128 heads’ worth of keys and values, and matrix absorption means it never has to decompress them. We derive it step by step: joint compression, absorption, why RoPE breaks it, and the decoupled RoPE branch that fixes it.https://zhangchitc.github.io/posts/rope/Sun, 13 Sep 2026 16:34:00 +0800https://zhangchitc.github.io/posts/rope/Sinusoidal encoding has a neat rotation property, but it breaks once the encoding goes through attention’s projection matrices. We expand the dot product to see why, then put RoPE through the same expansion and watch the absolute positions cancel out.