Your KV Cache Doesn’t Have a Bit Problem. It Has a Geometry Problem.
At identical 2-bit precision, one decision about which axis you quantize along swings a benchmark score from 2.88 to 63.53. Keys and values need opposite treatment — and the reason is in the attention equation, not the hardware. Take Llama-2-13B. Group its key-value cache by a quantization group size of 32 into two bits, while…