[Feature]: Add TurboQuant Support for KV Cache Quantization

# [Feature]: Add TurboQuant Support for KV Cache Quantization ## 🚀 The feature, motivation and pitch **TurboQuant** is a novel online vector quantization method that achieves near-optimal distortion rates for both MSE and inner product preservation, specifically designed for KV cache compression in LLMs. As described in the paper [TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate](https://arxiv.org/pdf/2504.19874), this method provides: - **Provably optimal distortion bounds** within a factor of ~2.7 of information-theoretic limits - **Unbiased inner product estimation** - critical for attention mechanisms - **Online application** - no data-dependent preprocessing required - **Accelerator-friendly design** - suitable for real-time AI workloads - **Significant memory savings** - 4-5× compression while maintaining model performance The paper demonstrates TurboQuant achieving perfect recall on Needle-in-a-Haystack tests at 4× compression and competitive performance on LongBench with just 2.5-3.5 bits per dimension. This would be a valuable addition to vLLM's quantization portfolio, complementing existing scalar methods (FP8, INT4, etc.) with a vector quantization approach specifically optimized for attention KV caches. ## Alternatives Current vLLM KV cache quantization options include: - **FP8 quantization** (`fp8_e4m3`, `fp8_e5m2`) - scalar quantization with element-wise MSE optimization - **Compressed-tensors KV cache** - supports various schemes

Posted by tunglinwood on Mar 26, 2026, 1:24 AM

Tags

• support (Other)

Related issues

• some bike vendor told me that the biggest problem with regular to ebike conversion was that regular bike frames were not strong enough to...
• I think of this as the kiosk economy.
• What a goofy comment.
• There's lots of evidence for positive influences of Caffeine on the body (some negative as well), especially the brain.
• Compose support in WSLc
• [RFC] Roadmap to v2.0