Iceberg Attention

New preprint: Iceberg Attention: Sublinear Long-Context Decoding with Mixture-of-Gaussians. At 1M tokens, MoG decoding is 27× faster than vLLM with FlashAttention, while matching exact attention and reading less than 5.3% of the KV cache.