What Is KV Cache?
KV cache is GPU's memory for every token that it saw in the conversation. It is stored as key/value pairs. It is there because model can look back at history. Every new token gets added to this cache. At 1 million tokens, that cache can get huge and create problems.
Why This Project Exists
Now here, I am not going to discuss which architecture and explain it. This project was done to actually show how important efficiency of KV cache is for modern LLMs, where most of our tasks hit 1 million tokens context, and agentic work requires high speed.
Model Size Is Not Enough
This project shows that model size is not enough. For instance, the model MiniMax M2.7 (230 Billion parameters and only 10 billion active parameters) is way smaller as compared to the massive Kimi K3 which has 2.8 Trillion parameters with 104 Billion active parameters.
Example:
Now Kimi K3 has almost 12× more total parameters and 10× more active parameters. But still at 1 million context, the total bytes read by the GPU per decode step for MiniMax M2.7 is 10% higher than the behemoth Kimi K3 — for M2.7, GPU read 256 GB but for K3 it read 233 GB at 1 million context.
Example:
Now to reinforce this example and give credit to DeepSeek: I would mention Qwen 3 600 Million parameters — it is pretty small. Yes, and now DeepSeek V4 Flash with total parameters being 284 Billion and Active parameters are 13 Billion which is roughly 22× more than Qwen 3 600 Million. But at 1 million context, the bytes read by GPU from memory for Qwen 3 600M is 233.33% higher than DeepSeek V4 Flash. So this shows that KV CACHE EFFICIENCY IS MORE IMPORTANT THAN MODEL SIZE.
The Formula
Every time a model generates one token, the GPU pulls two things from HBM (High Bandwidth Memory):
1. The active weights
Parameters actually used in that forward pass — to run the computation
2. The KV cache
Stored keys and values for every token generated so far — so attention can look back at history
Decode is memory-bandwidth-bound — the GPU's math units sit mostly idle waiting for data to arrive from HBM. So time per token is dictated by how many bytes get moved, not how many FLOPs get computed.
KV Cache Share (%)
KV_cache_bytes / (KV_cache_bytes + Active_weight_bytes) × 100
This is a traffic split, not a size comparison. It answers: of the total bytes streamed per token, what fraction is "remembering the past" vs. "running the model."
What It Means
Low % (near 0%) → the model is weight-bound. Almost all memory traffic is loading the model itself. Context length barely affects decode speed.
~50% → the tipping point. Cache and weights cost equally much to read per token.
High % (near 100%) → the model is cache-bound. The GPU is spending most of its memory bandwidth just re-reading conversation history, not the model. This is where long-context decode slows to a crawl regardless of how small or fast the model itself is.
KV Cache Share Is The Percentage of Total Bytes Read From Memory That Belong To KV Cache
Higher this number translates directly to lower throughput at large context. This shows efficiency of language model.
At 50% KV cache share, it means that GPU spends half the time fetching KV cache and the other half moving model weights. If it is less than 50%, the architecture is weight-bound — that is perfectly fine. But higher than 50% means the architecture and GPU are now KV cache-bound. A model with less KV cache share percentage is a great model.
Why Context Length Is the Hidden Variable
At short context, KV cache is tiny and weights dominate — the percentage sits near 0%. As context grows, KV cache grows linearly with tokens while weight bytes stay fixed. So this percentage isn't a static property of a model — it's a curve that climbs as context grows:
That is why we took context at 1 million tokens, which is the upper limit for almost all frontier models. Some models we tested at 1M context don't actually use that limit — but this was to show efficiency of the architecture used.
What "Relative decode speed ≈ 1 − KV Cache Share" Means
This formula estimates how much slower token generation gets as context increases, compared to short context.
The derivation
Starting from the core assumption that decode time per token is dominated by memory bandwidth:
Time_current_context ∝ Total_bytes_read = Weight_bytes + KV_cache_bytes
Time_current_context means the time taken to generate a single token at present context. At short context, KV cache is negligible, so the baseline time is essentially just weight-loading time:
Time_short_context ∝ Weight_bytes
Relative speed is baseline time divided by current time (faster = ratio closer to 1):
Relative speed = Time_short_context / Time_current_context = Weight_bytes / (Weight_bytes + KV_bytes)
Now simplifying it:
Weight_bytes / (Weight_bytes + KV_bytes)
= (Weight_bytes + KV_bytes − KV_bytes) / (Weight_bytes + KV_bytes)
= 1 − KV_bytes / (Weight_bytes + KV_bytes)
= 1 − KV Cache Share
That last term, KV_bytes / (Weight_bytes + KV_bytes), is the percentage we defined earlier. If cache is X% of your traffic, weights are the remaining (100−X)%, and that remaining share is your relative speed.
Concrete meaning
If KV Cache Share = 74% (Minimax M3 at 1M context):
Relative decode speed ≈ 1 − 0.74 = 0.26
Example:
Meaning: at 1M tokens of context, Minimax M3 generates tokens at roughly 26% of the speed it would achieve at near-zero context, purely because of memory traffic — the GPU is spending 74% of its time re-reading history instead of running the model.
Example:
On the other hand, the massive new model of Kimi K3 that has active parameters of 104 Billion, it has KV cache share of only 11% at 1M context, which means that it is a very efficient architecture and it will have relative speed of 89% at 1M context.
Example:
Now do not get started on DeepSeek V4 Pro — it is really special as it sits at 8.6%. Kimi does not get enough credit for their design. Also DeepSeek V4 Flash has 19.8% KV cache share at 1M context and MiMo V2.5 sits closer to it. To understand how efficient Kimi has made their architecture, compare K3's 11% share with Qwen 3.8 Max which has 94B active parameters and KV cache share of 31.6%.
Why this is a useful shortcut
Normally, predicting decode throughput requires knowing actual HBM bandwidth (GB/s), kernel efficiency, batch size, MTP, and much more. This formula gives you a relative slowdown factor using only a ratio of bytes, which cancels out hardware-specific constants.
Why Llama 70B Was Not Bad For Its Time
And what surprised me: Llama 70 Billion was not so bad for its time because it uses GQA such that there were 64 total query heads and there were 8 key_value heads. So there was ratio 1/8. So formula was this:
KV Cache Formula (GQA)
KV cache = tokens × layers × 2 × num_key_value_heads × head_dim × precision_bytes
Now Llama 3.1 8B has more KV cache share. Why? Because its ratio was 1/4 so there were 32 total query heads per layer and 8 key_value heads. Due to that ratio, KV cache share changed.
Layer Depth Invariance
For dense models, total params are proportional to layer count L:
params(L) = base_params_per_layer × L = C × L
And KV cache is also proportional to L:
KV_cache(L) = base_kv_per_layer × L = K × L
KV cache share:
KV_share(L) = KV_cache(L) / (KV_cache(L) + Active_params(L))
= (K × L) / (K × L + C × L)
= K × L / L(K + C)
= K / (K + C)
L cancels out completely. The share depends only on K/(K+C), which are per-layer constants (head_dim, num_kv_heads, precision), not on how many layers exist.
Numerical proof with Llama 3.3 70B:
| Layers | KV Cache |
|---|
| 80 | 305.2 GB |
| 40 | 152.6 GB |
| 20 | 76.3 GB |
| 10 | 38.1 GB |
Same ratio at every layer count.