llama.cpp Silent CPU Fallback: Why Quantized KV Cache Made Prefill 30x Slower (and the Fix)

The Symptom: You Saved VRAM and Prefill Died

You follow the standard advice: quantize your KV cache to free up VRAM on a small GPU. You restart llama-server with --cache-type-k q4_1 --cache-type-v q8_0, watch nvidia-smi confirm the model still fits on the card, and start using it. Generation speed looks fine. But the moment you send a real prompt — a document to summarize, a codebase chunk, a long chat history — it takes minutes before the first token appears, where before it took seconds.

Prompt processing (prefill) that used to run at 1,000+ tokens per second now crawls at 34–100 tokens per second. The longer the prompt, the worse it gets: a 90,000-token input that used to process in under two minutes now blows past the 600-second request timeout entirely. And there is no error message, no red text, no stack trace telling you what went wrong.

This is not your model, not your prompt, and not a driver issue. It is a silent backend fallback inside llama.cpp: when the CUDA flash-attention kernel cannot handle the KV cache types you requested, it quietly moves the entire attention operation to the CPU — and CPU-based attention is tens of times slower.

Diagnosis: Confirm It Is the Fallback, Not the Model

The telltale signature is the asymmetry between prefill and generation. In the report that first isolated this behavior (llama.cpp issue #27109), a Qwen 3.5 hybrid model on an RTX 3090 showed this split:

• q8_0 / q8_0 KV cache: 991–1276 t/s prefill, ~60 t/s generation.
• K=q4_1, V=q8_0: 34–106 t/s prefill, ~60 t/s generation.

Notice what did not change: generation speed stayed identical. Only prefill collapsed. That single observation is your best diagnostic — if per-token decode is unchanged but prompt processing fell off a cliff, you are almost certainly looking at a backend fallback, not a memory-pressure or thermal problem.

Two more confirmation steps, both free:

1. Watch GPU utilization during prefill. Run nvidia-smi dmon in a second terminal while you submit a large prompt with cache_prompt=false. If the card sits near 0% utilization while the prompt is supposedly processing, the work has been moved off the GPU.
2. Flip back to symmetric q8_0. Restart with --cache-type-k q8_0 --cache-type-v q8_0 and resend the same prompt. If speed instantly returns to 1,000+ t/s, the quantized type you picked is the trigger — and you were never constrained by the model at all.

Root Cause: Kernel Selection Fails Silently

The mechanism sits in the CUDA flash-attention kernel selection logic in ggml/src/ggml-cuda/fattn.cu. When llama.cpp tries to run attention with a quantized KV cache, it calls ggml_cuda_fattn_kv_type_supported() to decide whether a compiled kernel can serve that type. In a default build, that check is gated behind a compile-time flag that is OFF:

case GGML_TYPE_Q4_1:
case GGML_TYPE_Q5_0:
case GGML_TYPE_Q5_1:
#ifndef GGML_CUDA_FA_ALL_QUANTS
    return false;  // unsupported in default build
#endif

The default code path also rejects mixed K and V types — a K=q4_0, V=q8_0 combination fails a second guard that requires K->type == V->type. When either guard trips, the kernel selector returns BEST_FATTN_KERNEL_NONE. The graph scheduler then places the whole FLASH_ATTN_EXT operation on the CPU backend.

The critical detail: there is no warning and no offload log message. The scheduler handles it as a normal routing decision, so you get none of the usual “layer offloaded to CPU” output that would tip you off. The first symptom you ever see is the slowdown itself, which is why this bug is so hard to diagnose from the outside.

This is worth distinguishing from the normal dequantization cost. Even on a good build, reading a 4-bit KV cache costs some CPU-time to dequantize back to f16 — but that penalty is a few percent at ordinary context. The collapse documented here is not that. It is the difference between a dequant tax of a few points and an entire 20–30x drop from running attention itself on the CPU.

The Fix: Rebuild With GGML_CUDA_FA_ALL_QUANTS=ON

Because the missing kernels are a compile-time omission, the real fix is to rebuild llama.cpp with the extra kernels enabled. From the llama.cpp source directory:

cmake -S . -B build -DGGML_CUDA_FA_ALL_QUANTS=ON
cmake --build build --config Release -j

This compiles the additional flash-attention quant kernels (q4_1, q5_0, q5_1) and lifts the K == V restriction, so mixed configurations like K=q4_1, V=q8_0 now stay on the GPU. The verification table from issue #27109 is stark — the same RTX 3090, same model, same flag, only the build changed:

The underlying fix also matured into PR #27140, which added dedicated q4_0/q4_1/q5_0/q5_1 → f16 dequantization kernels so that all small KV quant types run within roughly 2% of q8_0 at equal prompt length. If you are on a recent build, check whether that fix has already landed — but do not assume: the guard flag is the only reliable signal.

Common Pitfalls That Mask or Mimic the Fallback

Four traps make this bug worse in practice, because each one sends you looking in the wrong place.

• Assuming the MMQ guard is involved. An older, related issue pointed at a shared-memory guard in the matrix-multiplication path. On many cards that guard passes fine and is not the trigger — issue #27109 confirms it explicitly. Do not pin this on the wrong mechanism.
• Running asymmetric K/V as a “quality boost.” People set --cache-type-k q4_0 --cache-type-v f16 thinking the f16 values buy quality at a small VRAM cost. On a default build this asymmetrical mix silently misses the fast fused path, as documented for AMD HIP in discussion #22411. The VRAM “saving” is real but you also pay the speed penalty — without knowing it.
• Trusting the config instead of measuring. Quantized KV cache can silently fall back to f16 on unsupported architectures. The flag you set is a request, not a guarantee. The only honest check is to load at a fixed context, record VRAM, change the setting, and confirm the number moved.
• Expecting a speedup. Even on a correct build, quantized KV cache is a memory feature, not a speed feature. f16 KV cache is actually faster whenever it fits. Quantize only when f16 genuinely does not fit at the context you need.

Verification: Prove the Fallback Is Gone

After rebuilding, verify rather than assume. Run the same long prompt with cache_prompt=false and compare against your baseline:

1. Record prefill tokens-per-second with the new build and your quantized KV flags.
2. Confirm it is in the same order of magnitude as your q8_0 baseline — not 30x lower.
3. Watch nvidia-smi dmon: the GPU should be pegged during prefill, not idle.
4. Check the server log for offload messages. A clean GPU path prints no CPU-fallback “offloaded layer” lines for attention.

For everyday guidance, the same community trail that diagnosed #27109 lands on a simple ladder: use q8_0 / q8_0 as the safe default on mainline builds. It halves KV cache memory with measurable-but-small quality loss and stays on the fast path everywhere (issue #24485 makes the case for making the extra kernels the default outright). Reach for q4_0 — and mixed K/V — only after rebuilding with the flag, and only when you have measured the VRAM saving actually solves your problem.

Conclusion

If you have already read our guide on fixing slow local LLM chats with KV cache offload and flash attention, this is the deeper trap that guide only gestures at: the same KV cache, when quantized, can silently fall off the GPU entirely. The signature is a prefill collapse with no error and unchanged generation speed. The fix is a rebuild with GGML_CUDA_FA_ALL_QUANTS=ON, verified by watching prefill throughput and GPU utilization return to baseline. Before you conclude your GPU is too small or your model is broken, check whether you are paying for an invisible CPU fallback — it is the cheapest 30x speedup most people will ever find.