How to Calculate KV Cache Size: A Step-by-Step Sizing Method

A GPU memory estimate based only on model weights leaves out the space needed to serve requests. Increasing context length or concurrency adds KV cache memory, and using the wrong head count can put the estimate several times above the actual requirement.

This guide calculates KV cache size from a model’s configuration, sequence length, batch size, and precision, with worked examples for four models. It also identifies architectures that need a different method. For help estimating costs for your workload, chat with our team.

The formula, and the two terms most versions of it drop

A complete KV cache estimate includes sequence length, batch size, and precision.

NVIDIA’s inference optimization guide gives a per-token formula for common attention architectures: Size of KV cache per token in bytes = 2 * (num_layers) * (num_heads * dim_head) * precision_in_bytes. The factor of two covers keys and values; multiply the per-token result across sequence length and batch size for the total.

Hugging Face’s Transformers documentation expresses the cache size as a count of floating-point values: twice the sequence length, head count, head dimension, and layer count multiplied together.

The floating-point count describes how cache size scales. To calculate bytes, include each element’s precision and the batch size, plus the factor of two for separate keys and values.

KV cache per token (bytes) = 2 * num_layers * num_kv_heads * head_dim * bytes_per_element
Total KV cache (bytes)     = per_token_bytes * sequence_length * batch_size

For grouped-query attention, use the number of KV heads rather than query heads.

Count KV heads, not query heads

Both formulas use attention-head count. NVIDIA notes that head count multiplied by head dimension commonly equals the model’s hidden size. That identity assumes multi-head attention, where each query head has its own keys and values. It does not hold for grouped-query attention, which all four worked examples use.

MQA and GQA reduce cache memory by storing fewer key and value heads. Use the KV head count when applying the formula to those architectures.

If you run the query-head count on a model with a 4:1 group ratio, your answer will be four times too large. Use num_key_value_heads to avoid that overestimate. We cover why attention produces separate key and value projections in our KV cache and attention piece; we cover why the cache exists and grows at all in our KV cache guide.

Calculate it by hand

Four steps: read the config, turn it into bytes per token, multiply by sequence length and batch size, then convert into the unit you provision in. Qwen3-8B carries the worked example.

Step 1: read four values out of config.json

Open your model’s repository on Hugging Face and read config.json. Four fields carry the whole calculation:

Derive head_dim when the configuration omits it. Mistral-7B-Instruct-v0.3’s config has no head_dim field at all. It gives “hidden_size“: 4096 and “num_attention_heads“: 32, and you derive the head dimension as 4096 / 32 = 128. Plenty of older configs are like this, so any tool that requires head_dim will fail on models people actually serve.

The fourth row is a choice, not a lookup. torch_dtype tells you what the weights were released in, and both Qwen3 configs and the Mistral config say “torch_dtype“: “bfloat16”. What your engine stores the cache in is a separate decision: two bytes per element for BF16 or FP16, one byte for FP8 or INT8. Halving that term halves every number downstream, which is the entire mechanism covered in our KV cache quantization piece.

Step 2: turn those four values into bytes per token

Qwen3-8B’s config reads “num_hidden_layers“: 36, “num_key_value_heads“: 8, “head_dim“: 128 and “torch_dtype“: “bfloat16”. Note the ratio hiding in there: “num_attention_heads“: 32 against 8 KV heads is a 4:1 group.

Per token, at two bytes per element:

2 * 36 layers * 8 KV heads * 128 head_dim * 2 bytes = 147,456 bytes = 144 KiB

Step 3: multiply by sequence length and batch size

The config also gives “max_position_embeddings“: 40960, so a single request at the model’s full native context holds:

147,456 bytes * 40,960 tokens = 6,039,797,760 bytes = 5.625 GiB

That’s one user. Eight concurrent requests at full context is 45 GiB, on a card that also holds the weights. At FP8 (one byte per element), the same single request drops to 2.8125 GiB.

Step 4: convert into the unit you provision in

Finish in GiB, the unit GPU capacity is quoted in, with 1 KiB = 1,024 bytes and 1 GiB = 1,024 MiB. Laid out that way, both remaining terms are plain multipliers:

Qwen3-8B at 144 KiB per token, BF16. Every row is that per-token figure multiplied by the sequence length and the batch size.

Nothing curves. Doubling context doubles the cache, and doubling concurrency doubles it again, which is why a model that fits comfortably in a demo falls over the first time eight users arrive at once.

The same arithmetic on three more models

Qwen3-32B is the useful comparison because only two things change. Its config reads “num_hidden_layers“: 64, “num_key_value_heads“: 8, “head_dim“: 128, and “num_attention_heads“: 64 against the 8B’s 32. The KV head count is identical.

2 * 64 layers * 8 KV heads * 128 head_dim * 2 bytes = 262,144 bytes = 256 KiB per token
262,144 bytes * 40,960 tokens = 10,737,418,240 bytes = 10 GiB

Four times the parameters buys roughly 1.78x the KV cache per token, because 64 layers against 36 is the only term that moved. Doubling the query heads did nothing. Parameter count is a poor proxy for cache size; layer count and KV head count determine it.

Mistral-7B-Instruct-v0.3, with the derived head dimension, lands at 2 * 32 * 8 * 128 * 2 = 131,072 bytes, or 128 KiB per token. Across its “max_position_embeddings“: 32768 that’s exactly 4 GiB for one request.

Layer, head and dimension values read from each model’s published config.json. Context lengths are each config’s own max_position_embeddings. 1 KiB = 1,024 bytes.

Where architecture breaks the formula

GQA fits the formula when you use the KV head count. Sliding-window and multi-head latent attention require different calculations.

Half of gpt-oss-120b’s layers never grow

gpt-oss-120b breaks the flat formula in an instructive way. Its config gives “num_hidden_layers“: 36, “num_key_value_heads“: 8, and “head_dim“: 64, which is half the head dimension every other model on this page uses. Assume 128 out of habit, and you’re 2x high before you start.

Run the flat formula, and you get 2 * 36 * 8 * 64 * 2 = 73,728 bytes, or 72 KiB per token, which, across its “max_position_embeddings“: 131072, is 9 GiB for one request. That number is wrong, and the config says why. It includes “sliding_window“: 128 and a layer_types array with 36 entries that strictly alternates “sliding_attention” and “full_attention“. Eighteen layers attend over the full sequence. The other eighteen cap their key and value state at a 128-token window.

The eighteen full-attention layers are the half that scales with context:

2 * 18 layers * 8 KV heads * 64 head_dim * 2 bytes = 36,864 bytes = 36 KiB per token
36,864 bytes * 131,072 tokens = 4,831,838,208 bytes = 4.5 GiB

Actual allocation depends on how the serving engine manages hybrid attention layers. The model configuration gives a ceiling on the sliding half, which never exceeds 128 tokens of state per layer, or 36,864 * 128 = 4,718,592 bytes, about 4.5 MiB. the flat 36-layer formula roughly doubles the real growth term at long context.

The model this formula doesn’t fit at all

DeepSeek-V3’s config reads “num_attention_heads“: 128, “num_key_value_heads“: 128 and “num_hidden_layers“: 61. Read literally, that’s a multi-head attention model with 128 KV heads across 61 layers, and the formula above would return the largest KV cache of anything discussed here by a wide margin.

That reading is wrong, and the same config contains the evidence. It also carries “kv_lora_rank“: 512, “qk_nope_head_dim“: 128, “qk_rope_head_dim“: 64 and “v_head_dim“: 128. DeepSeek-V3 uses multi-head latent attention, which caches a compressed latent vector plus a decoupled rotary component rather than 128 full key and value heads per layer. The num_key_value_heads field is vestigial. Run the standard formula on it and you’ll overstate the cache by more than an order of magnitude.

This method does not derive an MLA per-token size. For a model-specific comparison, the DeepSeek-V2 paper reports that against DeepSeek 67B, V2 reduces KV cache size by 93.3%. That is one model pair in one paper, not a property of MLA in general and not a figure for V3. If your model uses latent attention, use a sizing method specific to that architecture.

Five ways this calculation goes wrong

The query-head substitution is the big one, and it’s worth seeing the magnitude. Mistral-7B-Instruct-v0.3 at 32,768 tokens is 4 GiB using its 8 KV heads. Using its 32 query heads instead gives 2 * 32 * 32 * 128 * 2 * 32,768 = 17,179,869,184 bytes, or 16 GiB. Same model, same context, four times the provisioning.

The second is dropping batch size, which the four-factor phrasing invites. A per-request number is not a deployment number: Qwen3-8B goes from 5.625 GiB to 45 GiB across eight concurrent full-context requests, with nothing about the model changing.

Third is assuming two bytes per element when the engine is serving FP8. Fourth is assuming a head dimension of 128 because most models use it, which is exactly 2x wrong on gpt-oss-120b. Fifth is treating a hybrid attention stack as if every layer were global, or an MLA model as if it were GQA.

Every one of these is a wrong input, not wrong arithmetic. Getting the four numbers right is the entire job.

What your serving engine actually allocates

The number you just computed is a demand figure. It isn’t what vLLM hands the cache, and the difference trips people who size on paper and then hit out-of-memory errors anyway.

vLLM’s –gpu-memory-utilization caps the fraction of GPU memory assigned to the model executor, from 0 to 1, with a documented default of 0.92. It covers the whole executor, not just the cache. During startup profiling, vLLM allocates the remaining budget to KV state after weights, activations and CUDA graphs. Set –kv-cache-memory-bytes for an explicit cache budget; that setting overrides gpu_memory_utilization.

–max-model-len sets the combined prompt and output context length, defaulting to the model configuration. Its suffix parsing is case-sensitive: 1k means 1,000 while 1K means 1,024. The auto setting selects the maximum context length that fits in GPU memory.

Then read the startup log. vLLM prints lines like:

INFO 07-23 13:56:04 [kv_cache_utils.py:775] GPU KV cache size: 643,232 tokens
INFO 07-23 13:56:04 [kv_cache_utils.py:779] Maximum concurrency for 40,960 tokens per request: 15.70x

GPU KV cache size reports token capacity, while Maximum concurrency estimates how many requests fit at the stated per-request length. Compare that measured token capacity with your memory estimate at the same precision and workload settings.

When KV memory cannot accommodate a batch, vLLM may preempt requests and recompute them once space becomes available. Its documented remedies are to increase gpu_memory_utilization, reduce max_num_seqs or max_num_batched_tokens, or distribute the model across more GPUs.

Applying the sizing estimate

Four numbers from config.json, three you choose, four multiplications. Read num_key_value_heads rather than num_attention_heads, derive head_dim from hidden_size / num_attention_heads when the field is missing, multiply by batch, and check layer_types before you assume every layer grows. If your model uses latent attention, stop and go read its architecture paper instead.

What the number doesn’t tell you

The figure you just computed is what you provision. What you pay is a different quantity, because a lot of that cache holds context you’ve already processed at least once: the same system prompt, the same retrieved documents, the same agent scratchpad on every loop iteration.

Caching and reuse don’t shrink the number this page computed. They change how often you pay to build it, which is the argument for keeping the cache across requests rather than rebuilding it. We run that reuse in Tensormesh Platform, a self-hosted layer on top of the open-source LMCache project; our separate Serverless Inference service does not charge for cached tokens.

Tensormesh Platform supports vLLM. Our compatibility matrix provides model and configuration details.

Once you know your memory footprint, chat with our team about deployment costs for your workload. Or read the case for treating the KV cache as memory rather than cache.

Frequently asked questions

What is the KV cache formula?

2 * num_layers * num_kv_heads * head_dim * sequence_length * batch_size * bytes_per_element. The leading 2 accounts for the separate key and value tensors. NVIDIA publishes the per-token half of it, and Hugging Face publishes the scaling relationship, but neither writes it with both batch size and precision included.

How do I find num_kv_heads for my model?

It’s the num_key_value_heads field in config.json in the model’s Hugging Face repository. If a repository is gated behind an access request, don’t substitute a value from a third-party calculator or from memory. Get access and read the file.

How does GQA reduce KV cache size?

Grouped-query attention projects keys and values to fewer heads than there are query heads, so fewer key and value tensors get stored per layer. A model with 32 query heads and 8 KV heads stores a quarter of what the same shape would under plain multi-head attention.

Does this calculation cover FP8 or INT4 quantization?

It covers two bytes per element for BF16 and FP16 and one byte for FP8 and INT8, through the bytes_per_element term. Sub-byte formats are another matter. vLLM’s –kv-cache-dtype accepts int4_per_token_head, nvfp4, and several TurboQuant presets, but their effective storage per element depends on packing, scaling metadata and layout. Read the figure your engine reports rather than assuming half a byte. What any of it costs in accuracy is a separate question with no general answer.

Why isn’t DeepSeek-V3 used as an example?

Because its num_key_value_heads value of 128 is vestigial under multi-head latent attention, applying the standard formula overstates the cache by more than an order of magnitude. Use an MLA-specific sizing method instead.