Why Your Local LLM Slows to a Crawl on Long Chats: Fixing llama.cpp KV Cache Offload and Flash Attention

You load a quantized model, fire up llama-server or llama-cli, and the first few turns feel snappy. Then something strange happens: by the time the conversation reaches a few thousand tokens, generation crawls to a fraction of its original speed, memory usage climbs, and every prompt takes a painfully long “prefill” burst before the first token appears.

This is not a broken model, and it is not your GPU degrading. In the overwhelming majority of cases it is the KV cache—the growing attention state that stores every token already generated—combined with how you have (or have not) told llama.cpp to offload it and accelerate its computation. This article walks through the symptoms, the diagnosis, the exact flags to change, and how to verify the fix without guessing.

The Symptom: Fast Start, Then a Slow Crawl

The classic signature is easy to recognize once you know what to look for:

• Short prompts are fast. A single-turn query returns quickly because the KV cache is small and cheap to process.
• Long chats degrade. As the history grows, prompt processing (“prefill”) takes longer before the first generated token, and per-token generation drops.
• RAM usage grows. The KV cache scales linearly with context length, and if it has spilled off the GPU it is being read from system RAM on every decode step.
• The model eventually “forgets” or truncates. When the cache reaches the configured context size, llama.cpp triggers context shift: it drops the oldest tokens and reprocesses the remainder.

Any one of those is a signal, but the combination of growing memory plus longer prefill plus slower decode points squarely at KV cache management rather than at the model weights themselves.

Diagnosis: Separate Model VRAM from KV Cache

Before changing any flag, confirm where your memory is actually going. Run your server with the default settings and watch both GPU and system memory during a long session. On Linux use nvidia-smi for VRAM and free -h for RAM; if you are running the server under a manager such as Ollama, it exposes similar numbers in its own logs.

Two distinct regions matter here, because llama.cpp treats them differently:

• Model weights. The quantized tensors that get partially or fully offloaded to the GPU via --n-gpu-layers (short form -ngl). These are mostly static during inference.
• KV cache. The key and value tensors that grow with every token in the active context. Their size depends on context size, model dimension, and the KV data type you allow.

The single most common misconfiguration is loading the weights onto the GPU but leaving the KV cache — by default — unquantized and potentially split awkwardly across devices. By default the KV cache is stored at f16, which for a large context can consume more memory than the model weights themselves. Understanding this split is what turns the fix from trial-and-error into a deliberate configuration.

The Fix: Tune Offload, Flash Attention, and Context Shift

Here are the levers that matter, in order of impact.

1. Put the KV cache in the right device

-ngl / --n-gpu-layers controls how many of the model’s layers are offloaded to the GPU. The KV cache placement is tied to where the relevant layers live. A common “slow long-context” failure is loading only some layers to the GPU so that the KV cache ends up straddling GPU and CPU, forcing data movement on every decode step. If you want the KV cache to stay on the GPU, offload all layers (-ngl 99 or higher) so the K/V tensors are not split.

The inverse is also useful on memory-constrained cards: -ot / --no-offload-kqv keeps the K, Q, and V tensors on the CPU while the rest of the model runs on the GPU. This trades speed for lower VRAM and is a legitimate choice when you are trying to fit a large model into a small card with a long context.

2. Quantize the KV cache to shrink it

The single highest-leverage change for long-context degradation is quantizing the KV cache itself. llama.cpp exposes --cache-type-k and --cache-type-v (in some releases combined into a single --cache-type). Allowed values include f16, q8_0, and q4_0. Moving from f16 to q8_0 roughly halves the KV cache memory for a modest quality cost; q4_0 reduces it further with a larger quality trade-off.

This matters precisely because the KV cache, not the weights, is what balloons during a long chat. Quantizing it lets you hold a bigger context in the same VRAM, which postpones the moment context shift kicks in.

3. Enable Flash Attention for the attention math

-fa / --flash-attn switches attention to a fused Flash Attention kernel. On supported hardware this speeds up both prefill and decode and, importantly, lowers the memory footprint of the attention computation. If your GPU supports it, this is almost always a strict win for the long-context scenario that is causing the slowdown. If the build was not compiled with it, the flag will report as unavailable rather than silently failing.

4. Control context length and mmap behavior

• -c / --ctx-size sets the maximum tokens of history the KV cache can hold. Setting it too high wastes VRAM reserved up front; setting it too low causes premature truncation. Match it to your actual use (for example 4096 or 8192 for RAG, larger only if you genuinely need it).
• --no-mmap disables memory-mapping of the model file and instead loads it fully into RAM. This can improve stability on some systems at the cost of higher startup RAM and longer load time, but it does not affect the KV cache directly—apply it only if you observe page-cache thrashing or load-time issues.
• Context shift / --no-context-shift controls what happens when the cache fills. By default llama.cpp shifts the context (drops the oldest tokens and re-processes). This re-processing is one of the visible “stalls” users describe. Disabling context shift forces a full recompute of the entire history, which is almost always worse; instead, reduce the likelihood of hitting the limit by quantizing the cache and sizing the context correctly.

Common Pitfalls That Look Like This Bug but Are Not

Before you conclude it is the KV cache, rule out a few lookalikes:

• Thermal throttling. A card that boosts for the first minute and then throttles under sustained load produces the same “fast then slow” curve. Check nvidia-smi clocks and temperature before blaming the cache.
• Model weights partly on CPU. If you set -ngl too low, every layer left on the CPU serializes the pipeline. This is a weights-offload problem, not a KV cache problem, though the two are often confused because both are fixed by raising -ngl.
• Context size grossly oversized. Reserving a 32K context on an 8GB card leaves little room for anything else and can cause the GPU to spill or the OS to swap, both of which look like slow degradation.
• Concurrent workloads. If another process—say a ComfyUI image pipeline—is competing for the same 16 GB card, both will degrade. Always confirm GPU exclusivity when benchmarking.

Verifying the Fix

Change one flag at a time and measure, so you know which change actually helped:

1. Run a fixed multi-turn prompt of a known token length with your baseline flags and record the prefill time and tokens-per-second shown in the server log.
2. Apply --cache-type-k q8_0 --cache-type-v q8_0 and re-run the identical prompt. Note the change in decode speed and the lower VRAM reported by nvidia-smi.
3. Add -fa and re-run. Confirm the flag was accepted (the log states whether Flash Attention is active) and note the prefill improvement.
4. Finally, load a genuinely long context near your -c limit and confirm the server no longer stalls on context shift the same way.

If decode speed improves but prefill is still slow, you have fixed the cache but still need better attention (step 3). If memory drops but speed does not improve, your bottleneck is elsewhere—return to the pitfalls list and re-check clocks and layer offload.

Putting It Together: A Practical Starting Point

A reasonable baseline for a 16 GB card running a ~14B quantized model with a moderate context looks like this (flags shown as llama.cpp accepts them):

llama-server -m models/model-Q8_0.gguf \
  -ngl 99 \
  -c 8192 \
  -fa \
  --cache-type-k q8_0 \
  --cache-type-v q8_0

Adjust -ngl down (and consider --no-offload-kqv) only if you hit CUDA out of memory during load or when the context grows. Adjust the context size down if you still see stalls. The point is not any single magic value; it is understanding that the long-conversation slowdown is a cache problem, and that the cache is something you can deliberately size, quantize, and accelerate.

Conclusion

When a local model starts fast and ends slow, the weights are almost never the culprit—the KV cache is. By separating model memory from cache memory, quantizing the cache, enabling Flash Attention, and right-sizing your context, you can hold longer conversations without the telltale crawl. Make one change at a time, measure with real logs, and you will know exactly which lever fixed it.

For more on the lower-level mechanics of quantized models and GGUF loading, see our guide on llama.cpp CUDA out-of-memory and layer offload, and for tuning ComfyUI’s own GPU memory under the same constrained card, see our guide to ComfyUI VRAM launch flags.

Primary Documentation

For the current, authoritative definitions of these flags—including --cache-type-k/v, --flash-attn, --n-gpu-layers, and context-shift behavior—consult the official llama.cpp repository and documentation. Flag names and defaults change between releases, so verify them against the exact version you have installed.