KV Cache Quantization: FP8, FP4, and What They Cost You

As context lengths and concurrent requests grow, the KV cache can consume more GPU memory than model weights. Reducing its precision frees memory for longer contexts or more requests, but the savings come with tradeoffs in accuracy, compatibility, and runtime overhead.

FP8 and FP4 are floating-point formats that use eight and four bits per value, respectively. In KV cache quantization, they represent the cached key and value tensors at lower precision than common 16-bit formats such as FP16 or BF16. The model’s weights stay unchanged. Fewer bits reduce the memory needed for cached values, although scaling metadata adds some overhead.

That extra capacity can help a serving system handle more requests at once or retain longer contexts. It does not guarantee faster responses or unchanged output quality. Results depend on the model, the workload, how you scale the values, and whether the serving engine can process the quantized cache efficiently.

FP8 and FP4 make different compromises. This guide compares their memory and accuracy tradeoffs, explains calibration and configuration across vLLM, SGLang, and TensorRT-LLM, and shows what to measure before enabling either format in production.

What quantization changes, and what it doesn’t

KV cache quantization stores the key and value tensors your attention layers already produced in a lower-precision format, so more tokens fit in the same GPU memory. It changes the cache, not the model. If you need the cache explained first, start with our KV cache guide; the argument for caring about its size is that the KV cache is really memory, not a speed trick.

That distinction gets blurred constantly. Model-weight quantization (FP8 weights, AWQ, GPTQ, W4A16) changes what the model is; KV cache quantization changes only what you keep after a token is processed. In TensorRT-LLM’s precision documentation, its per-model support matrix covers weights and activations, and only two flags there touch the cache.

FP8 reduces KV cache memory use, leaving room for more tokens and potentially supporting higher throughput or longer contexts. vLLM‘s feature page describes these capacity benefits but does not quantify an accuracy penalty.

The flag everyone shows you is the uncalibrated one

Quantization needs scaling factors: numbers mapping the original tensor range onto the smaller format’s range. vLLM documents three ways to get them, and the choice is the most consequential decision in this feature.

vllm serve <model> --kv-cache-dtype fp8

Without calibration, kv_cache_dtype=”fp8″ uses scaling factors of 1.0. vLLM recommends estimating scales from a representative dataset with llm-compressor. This calibrated path also supports per-attention-head quantization, currently limited to the Flash Attention backend.

SGLang loads scaling factors from the checkpoint and falls back to 1.0 when none are supplied, which may reduce accuracy. FP4 handles its scaling factors dynamically during quantization and dequantization.

If you benchmarked FP8 KV cache once, disliked the result, and moved on, check which of these you ran. Uncalibrated and calibrated FP8 are different configurations, and only one is recommended.

FP8, FP4, and where INT8 actually sits

FP8 formats trade exponent bits for mantissa bits. SGLang lists E5M2 with a wider range of ±57,344 and lower precision, while E4M3 offers higher precision within ±240. In vLLM, FP8 resolves to E4M3 on CUDA and ROCm. SGLang generally recommends E4M3 for better accuracy.

SGLang’s FP4 support is experimental. Its E2M1 implementation uses block-based microscaling: groups of 16 consecutive elements share an 8-bit exponential scale, compared with the 32-element blocks in the OCP specification.

Scaling metadata reduces the memory savings below the raw bit-width ratio. SGLang estimates that FP4 holds about 3.56 times as many tokens as BF16 and 1.78 times as many as FP8, after accounting for scaling overhead.

Lower bit width reduces nominal KV payload. Effective cache capacity also depends on scaling metadata, format layout, and implementation.

Capacity example: SGLang quantized KV cache documentation.

INT8 support varies by engine: vLLM’s engine args accept int8_per_token_head, and TensorRT-LLM has an INT8_KV_CACHE build flag. SGLang’s flag carries no INT8 value, and none of the three publishes an accuracy result for it. Confirm support and evaluate accuracy before using it.

Check that the attention backend supports quantized KV cache and fuses dequantization into the attention kernel. SGLang warns that a separate dequantization step can be extremely slow, offsetting the throughput benefit of lower memory use.

Keys and values don’t want the same treatment

Quantization granularity determines how many scaling factors are stored. TensorRT-LLM defines per-tensor scaling as one factor for the entire tensor, per-channel as one factor per channel, and per-token as one per token. These definitions apply to tensor quantization generally, not only to KV caches.

vLLM supports per-tensor FP8 KV scaling, with separate scales for Q, K, and V, and per-attention-head scaling with q_scale = [num_heads] and k/v_scale = [num_kv_heads]. SGLang currently supports only per-tensor scalar scales.

KIVI, an ICML 2024 paper, found that keys benefit from per-channel quantization while values benefit from per-token quantization. Their different element distributions make a single treatment less effective. LMCache’s TurboQuant presets likewise use different widths for keys and values, while KVQuant also begins with per-channel key quantization.

Some layers break before the others do

Some attention layers, including sliding-window layers, are more sensitive to KV quantization. vLLM provides –kv-cache-dtype-skip-layers to leave named layers at the model’s native dtype. It takes layer indices or attention type names, and defaults to an empty list:

vllm serve <model> \  
  --kv-cache-dtype fp8 \  
  --kv-cache-dtype-skip-layers sliding_window

This matters more on hybrid models than the flag’s obscurity suggests. gpt-oss-120b’s config declares 36 layers whose layer_types array strictly alternates sliding_attention and full_attention: 18 of 36 are sliding-window, at a sliding_window of 128. The skip flag is half that stack.

Use the flag to evaluate sensitive layers on your model and workload; it does not imply a fixed accuracy improvement.

Accuracy depends on the model and benchmark

Pick your format against your own task mix, because the same model at the same bit width can lose almost nothing on one benchmark and half its score on another.

Accuracy impact is model- and format-dependent. vLLM’s feature page and TensorRT-LLM’s precision page do not provide a universal penalty. SGLang describes potential accuracy degradation, while LMCache identifies its FP8 serialization as lossy with a small, unquantified accuracy impact.

Every number that does exist is bound to a method, a model, a bit width, a task, and a calibration state. SGLang publishes the most useful set, labeled preliminary and sourced from two pull requests:

The same KV precision can behave very differently across tasks. These preliminary results support testing the intended workload, not assuming a universal quality penalty.

Preliminary SGLang results; accuracy varies by model, task, and configuration.

Compare the two GPT-OSS-120B rows. Same model, same FP4 bit width: gsm8k moves from 0.9161 to 0.9152, a small change, and aime25 falls from 0.7533 to 0.3533, more than half the score. Testing only gsm8k would miss the larger regression on aime25.

SGLang reports that models above 200B parameters generally tolerate FP4 better than smaller models. Its Qwen3-235B AIME25 result remains at 0.6000, whereas GPT-OSS-120B declines sharply. It also warns that degradation may be greater with long contexts.

FP8 is not free on this data, and it is not uniformly costly either. Qwen3-235B’s aime25 drops from 0.7733 to 0.7333 and its gpqa_diamond from 0.7010 to 0.6899, while GPT-OSS-120B moves the other way on both: 0.7533 to 0.7667 on aime25, 0.5081 to 0.5434 on gpqa_diamond. The FP8 movements in these preliminary runs are small and mixed in direction, suggesting neither losslessness nor a universal quality penalty.

KVQuant reports less than 0.1 perplexity degradation at 3-bit quantization on WikiText-2 and C4 using its four-technique method on LLaMA, Llama-2, Llama-3 and Mistral. KIVI reports nearly unchanged quality with 2-bit quantization on Llama, Falcon, and Mistral while reducing peak memory by a factor of 2.6. That memory figure includes model weights, not just the KV cache.

Evaluate FP4 on your own model and representative workload before deployment.

Engine support and configuration

The engine-arguments reference lists the available values for –kv-cache-dtype: “auto, bfloat16, float16, fp8, fp8_ds_mla, fp8_e4m3, fp8_e5m2, fp8_inc, fp8_per_token_head, int4_per_token_head, int8_per_token_head, nvfp4, nvfp4_4over6, turboquant_3bit_nc, turboquant_4bit_nc, turboquant_k3v4_nc, turboquant_k8v4.” Build your support matrix from the engine args.

In LMCache’s L2 serde options, turboquant_k8v4, turboquant_4bit_nc, turboquant_k3v4_nc and turboquant_3bit_nc control storage serialization. Evaluate their storage and accuracy tradeoffs separately from the serving engine’s live KV cache format.

LMCache serialization acts on a different part of the cache: data moving to and from an L2 adapter. It compresses disk or remote-storage data rather than the tensors used directly by GPU attention. FP8 serialization halves byte volume relative to BF16 with a reported small accuracy loss. TurboQuant presets range from the default turboquant_k8v4 (FP8 keys and 4-bit values) down to turboquant_3bit_nc. Storage compression can be combined with GPU KV quantization.

Tensormesh Platform manages a three-tier hierarchy: L0 in GPU HBM, L1 in host RAM and L2 in filesystem-backed storage.

We run LMCache as a self-hosted layer with vLLM. Our compatibility matrix provides model and configuration details. Our separate Serverless Inference service doesn’t charge for cached tokens, so context reuse can reduce the bill. The Amnesia Tax explains the cost of reprocessing context you already paid for.

Choosing a quantization configuration

Start with FP8 E4M3. vLLM and SGLang both expose E4M3 options explicitly, so it has the widest documented backend support of the three. TensorRT-LLM documents an FP8_KV_CACHE build flag rather than a named E4M3 option, so confirm its backend and recipe for your target model.

Then calibrate, because the uncalibrated default is what the docs warn about. Treat that calibration as belonging to a checkpoint and a workload, not as a one-time setup step: scales come from a representative calibration dataset, so recalculate them when the model, the checkpoint, or the representative traffic changes. Check that your attention backend fuses dequantization before you measure anything.

Then, and only then, look at four-bit. FP4 buys roughly 1.78x more tokens than FP8 by SGLang’s own figure, and in the same table it halved one model’s aime25 score. Evaluate that capacity gain against accuracy on your own tasks, especially competition math or long-context reasoning. Quantization is one lever, and our other cache-memory levers involve different trade-offs.

Contact our team to discuss KV cache memory use in your deployment.

Frequently asked questions

What is KV cache quantization?

Storing a model’s cached key and value tensors in a lower-precision format, such as FP8 or FP4, so each token takes fewer bytes and more fit in GPU memory. It changes only the cache. The model weights are untouched.

Does FP8 KV cache hurt accuracy?

No engine publishes a general figure. SGLang’s preliminary benchmarks show small movement in both directions at FP8: Qwen3-235B-A22B’s aime25 falls from 0.7733 to 0.7333, while GPT-OSS-120B’s rises from 0.7533 to 0.7667. Whether that matters is a question about your tasks, so measure it on your own evals.

Why is KV cache so large?

It grows with every token, for every layer, for every request in flight, and long contexts multiply all three. Our KV cache guide covers the sizing arithmetic.

How do I optimize KV cache memory?

Quantization is one lever, and this piece covers it. Our KV cache memory guide covers the others and how they compare.