llama-server Uses RAM Even With Free VRAM: The KV Cache Spill and How –cache-type-k/-v Fix It

You set --ctx-size 131072 on a model that fits comfortably in your GPU, launch llama-server, and open nvidia-smi expecting the card to be busy. Instead you see the VRAM half-empty while your system RAM is being eaten alive — tens of gigabytes of it. Generation still works, but it is slow, and other desktop programs start swapping. The model is on the GPU, so what is eating all that RAM?

In almost every case the answer is the KV cache. This article explains why it happens, how to confirm it on your own machine, and the two flags — --cache-type-k and --cache-type-v — that pull the cache back onto the card. It is the same class of problem reported repeatedly in the llama.cpp community: a user runs a 27B model at 128K context with q8_0 cache and watches RAM fill despite free VRAM.

The Symptom: VRAM Free, RAM Exploding

The tell-tale signature is a mismatch between your two memory monitors. Run nvidia-smi and free -h side by side during a long generation. If you see the GPU reporting only a fraction of its VRAM used while the process keeps growing in system RAM as the context fills, you are looking at a KV cache that has spilled off the GPU.

This is easy to misread. People assume that if the weights fit in VRAM, everything should stay on the card. But llama.cpp allocates three distinct tenants in memory: the model weights, a working buffer and activations, and the KV cache. Each is sized independently, and the KV cache scales with your context length even when the model is tiny.

A 7B model at 128K context can easily need more memory for its KV cache than for its weights. When that cache no longer fits in the VRAM left over after the weights, llama.cpp quietly offloads the remainder to CPU RAM over the PCIe bus. There is no error and no warning — you just get slower tokens and a bloated free -h.

Why the KV Cache Grows so Fast

The KV cache stores the key and value tensors for every token in the context so that later tokens can attend to earlier ones without recomputing them. Its size is linear in context length and in the hidden dimension, so doubling your context doubles the cache.

The default cache data type is f16, meaning each key and each value element occupies 2 bytes. The rough budget for a single token’s KV (both K and V) is:

bytes_per_token = 2 bytes × 2 (K and V) × n_layers × n_kv_heads × head_dim

KV_cache_bytes = bytes_per_token × context_length

Because the per-token figure is large, a long context balloons fast. For a modern 27B-class model this can be several hundred bytes per token, so a 128K context translates to tens of gigabytes in the KV cache alone — more than the remaining VRAM on a 16 GiB or even 24 GiB card after the weights load. The fix is not to shrink context; it is to make each cache element cheaper via quantization.

The Fix: Quantize K and V Separately

llama.cpp lets you quantize the key cache and the value cache independently:

./llama-server \
  -m model.gguf \
  -ngl 99 \
  --ctx-size 32768 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0

The allowed values for both flags are f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, and q5_1. The two most useful in practice are q8_0 and q4_0.

q8_0 stores each element in 8 bits, roughly halving the cache size relative to f16. q4_0 stores each element in 4 bits, cutting it to a quarter of the f16 footprint. For a cache that was spilling tens of gigabytes into RAM, switching from f16 to q8_0 is frequently enough to pull the entire cache back onto the GPU. When you need the full 128K context and the card is genuinely tight, q4_0 on the value side (--cache-type-v q4_0) is the heavier lever.

There is a real robustness argument for quantizing K and V differently. The key cache tolerates lower precision better than the value cache for many tasks, so a common tuned configuration is --cache-type-k q8_0 --cache-type-v q4_0 — mild quantization on K, aggressive on V. It is the asymmetric-KV pattern discussed in the llama.cpp community, and it preserves more quality than blanketing both sides in q4_0 at the same memory cost.

Verifying the Cache Actually Stays on the GPU

Flipping the flags is only half the job. You need to confirm the cache is now resident in VRAM rather than assuming the flags worked:

• Compare before and after. Run the same generation at the same context with f16 and then with q8_0. Watch nvidia-smi and free -h. The RAM footprint of the process should drop sharply, and VRAM should rise.
• Look at startup logs. llama-server prints an estimate of the KV cache size and where it will live at launch. If the estimated size exceeds remaining VRAM, part of it is heading to RAM.
• Check tokens-per-second. A cache that lives in VRAM is much faster than one paged over PCIe. If your t/s jumps after the change, the cache has moved on-card.
• Watch for quality regressions. q4_0 is aggressive enough to degrade some outputs. If you depend on long-context precision, benchmark q8_0 against q4_0 on your own prompts rather than trusting rules of thumb.

One subtlety worth keeping in mind: --ctx-size sets the maximum, but the cache is allocated to match it. If you rarely use 128K of context but leave the ceiling at 128K, you are reserving KV memory you never touch. Where the workload allows, lowering --ctx-size to something realistic is a free win that often removes the spill entirely without touching precision.

Common Pitfalls

Three mistakes account for most of the confusion around this topic. First, people set -ngl 99 and assume that forces everything onto the GPU. -ngl controls layer offload only; the KV cache is sized independently and will spill regardless of how many layers are on the card.

Second, people conflate weight quantization with cache quantization. A model quantized to Q4_K_M still uses an f16 KV cache by default. Quantizing the GGUF does nothing for the cache, which is why you must set --cache-type-k and --cache-type-v explicitly even on a heavily quantized model.

Third, people chase the spill with more RAM instead of less cache. Adding system RAM does not fix the PCIe bottleneck; it just delays the swap storm. The performance-correct fix is to keep the cache on the card, which means quantization or a smaller context — not a bigger RAM budget.

Conclusion

A llama-server that fills system RAM while VRAM sits unused is almost never a bug in your driver or your install. It is the KV cache being sized for a long context with a high-precision default and quietly overflowing the card. The levers you actually control are the context ceiling and the cache data types.

Start with --cache-type-k q8_0 --cache-type-v q8_0, confirm the RAM footprint drops and the card fills, and only reach for q4_0 on the value side if you need the longest contexts on a genuinely small card. Verify each change against nvidia-smi, the launch estimate, and your own output quality — the flags are free to try, but the result depends entirely on your specific model, context, and card.

For the broader memory model behind layer offload and mmap flags — a different failure signature from this one — see our earlier walkthrough of llama.cpp CUDA out of memory and –n-gpu-layers. The authoritative flag reference is the llama-server(1) man page, and the llama.cpp repository tracks ongoing cache-quantization work such as asymmetric K/V and iq4_nl.