Tag: Cuda

Few local-AI errors are as frustrating as CUDA out of memory. You load a GGUF model, bump --n-gpu-layers to “everything,” and watch the process die mid-context. The instinct is to blame the model or the GPU, when the real problem is usually a misunderstanding of how llama.cpp splits work between VRAM, system RAM, and disk-backed memory mapping.

This article is a documentation-based walkthrough of llama.cpp’s memory model. It explains what --n-gpu-layers, --mmap, --no-mmap, --mlock, and the KV cache flags actually do, how they interact, and how to pick values that fit a 16 GiB card. No fabricated benchmarks here — everything is drawn from the llama.cpp source and documentation, and the reasoning is reproducible on your own hardware.

Why the Out-of-Memory Error Is Misleading

The CUDA out of memory message reports a symptom, not a cause. It means a specific allocation request failed at a specific moment — often during KV cache expansion or when a new layer is pushed to the GPU. It does not tell you that your --n-gpu-layers value was too optimistic, or that your context length forced a KV allocation larger than the remaining VRAM.

The first diagnostic rule is simple: read the full error, including the allocation size and the free-vs-total figure on the line above it. A request to allocate a small chunk (say 224 MiB) that fails while the GPU reports only ~164 MiB free points to fragmentation and overcommitment, not a single giant allocation. This is the classic signature of a model that almost fits but does not, and it usually means your layer offload setting is one or two layers too high. When the request is large — several gigabytes — the cause is more fundamental: you asked to offload more layers than the card can hold.

The Three Tiers of llama.cpp Memory

llama.cpp manages model weights and activations across three physical tiers. Understanding them is the key to every fix below.

• VRAM (GPU memory): The fastest tier, used for layers offloaded with --n-gpu-layers and for the GPU-resident portion of the KV cache.
• System RAM (CPU memory): Holds whatever the GPU can’t. Layers not offloaded run here, and the CPU-resident KV cache lives here.
• Disk-backed mapping: With default --mmap enabled, llama.cpp memory-maps the GGUF file so that unneeded weight pages never occupy physical RAM; the OS pages them in on demand.

The core trade-off is explicit: every layer you move to the GPU with --n-gpu-layers buys speed but consumes VRAM, and every layer you leave on CPU saves VRAM but slows generation. For a Q4_K_M model around 7B parameters, the full weight footprint is roughly 4 GiB; a 14B is closer to 8 GiB. These approximate figures explain why a 16 GiB card can often hold a 7B model fully offloaded but only partially offload a 14B or larger.

What –n-gpu-layers Does (and Does Not) Do

--n-gpu-layers N tells the GGML backend how many of the model’s transformer layers to place on the GPU, counting from the output side. Setting -1 (or omitting it for CUDA builds) requests “as many as fit,” but that auto-detection has limits: it estimates space for the weights and then discovers the KV cache and activation buffers shrink the real budget. The result is the exact “almost fits” failure described above.

A reliable iterative approach is to start low and raise by hand. For a 14B model on 16 GiB, begin around --n-gpu-layers 20, confirm it starts, then raise in increments of 5 while watching nvidia-smi in a second terminal. Stop when the used VRAM settles at roughly 80% of capacity, leaving headroom for KV cache and activations. For a 7B model, --n-gpu-layers -1 is often safe; for anything larger, explicit values are almost always the right call. This is exactly the kind of troubleshooting regular readers will recognize from our earlier look at running large open models on consumer hardware, where offloading discipline is equally decisive.

–mmap, –no-mmap, and –mlock: The RAM Side of the Equation

These three flags control how seriously the process treats system RAM, and they matter most when your model is too big to fit in VRAM and you rely on CPU execution.

• --mmap (default): Memory-maps the GGUF file. Weight pages are loaded lazily from disk. This lets you run a model larger than physical RAM, at some cost to responsiveness as the OS pages.
• --no-mmap: Loads the file into memory eagerly instead of mapping it. Faster warm-up and more predictable speed, but it requires the full weight footprint to fit in RAM.
• --mlock: Locks the mapped pages into RAM so the OS cannot swap them out. Use it only when you have RAM to spare and want to avoid swap-induced stalls; on a machine that is already tight, it can trigger an immediate allocation failure.

A common configuration error is combining --mlock on a RAM-constrained host with aggressive GPU offload. Each flag alone is reasonable; together they can each request more memory than the system has, and the failure manifests as an OOM — sometimes from the OS killer, sometimes from CUDA — that is not obvious from the flags themselves. When debugging, strip the aggressive flags first and add them back one at a time.

Common Pitfalls That Produce OOM

These are the recurring failure patterns reported in the llama.cpp issue tracker and across deployment guides. Each has a concrete cause and a direct fix.

• Full offload plus a large context: --n-gpu-layers -1 --ctx-size 32768 looks reasonable until the KV cache for a 32K context claims gigabyte after gigabyte of VRAM. Fix: lower --ctx-size, or size the cache manually with --cache-size.
• Quantization mismatch: A Q8_0 or F16 file is two to four times larger than its Q4_K_M sibling. The same --n-gpu-layers value that works for Q4 will OOM on Q8. Fix: know your quantization’s footprint before choosing layer counts.
• Multiple processes sharing the GPU: A leftover process or a desktop compositor holding several hundred MiB is often the “last straw” behind an almost-fits failure. Fix: check nvidia-smi and close competing processes before raising layers.
• Ignoring the MoE layer behavior: On MoE models, expert weights follow their own offload path. Forcing all layers to GPU on an MoE model can OOM even when the same flag works on a dense model of similar advertised size.

A Reproducible Diagnostic Sequence

When a model OOMs, run this sequence in order. It isolates the cause without destroying your configuration.

1. Record the baseline: Run nvidia-smi and note free VRAM before launching. Subtract 200–400 MiB for driver overhead.
2. Start conservative: Launch with a low explicit --n-gpu-layers (e.g. 10) and a modest --ctx-size 4096. Confirm the process starts and produces tokens.
3. Raise incrementally: Increase layers in steps, watching nvidia-smi after each start. Watch specifically whether the increase is roughly linear — a sudden jump indicates the KV cache crossed a boundary, not the layers.
4. Add the KV flags last: Only after you settle on layers, widen --ctx-size or set --cache-size. The cache is frequently the hidden variable in OOM reports.
5. Reintroduce RAM flags: Save --mlock and --no-mmap for last, once VRAM is stable, and confirm RAM via free -h before enabling them.

This sequence deliberately separates the GPU-side variables (layers, cache) from the CPU-side variables (mapping, locking). Most OOM confusion comes from changing all of them at once and losing track of which one moved the needle.

Choosing Values That Actually Fit a 16 GiB Card

There is no universal magic number, but these starting points are reasonable for a 16 GiB card with a clean GPU. Treat them as hypotheses to verify, not prescriptions, since driver overhead and concurrent loads vary by machine.

• 7B Q4_K_M (~4 GiB weights): full offload (--n-gpu-layers -1) usually fits with room for a modest KV cache.
• 13B–14B Q4_K_M (~8 GiB weights): explicit partial offload, roughly half to two-thirds of layers, leaving real headroom for cache and activations.
• 30B–34B Q4_K_M (~18 GiB+ weights): CPU-and-RAM execution with --mmap is generally required; VRAM offload may be a small fraction or none at all.

These figures are documentation-derived estimates and should be checked against your own nvidia-smi and free -h output. The discipline is the point: match your offload level to a measured VRAM budget, not to a guess.

Verifying the Fix, Not Just the Absence of the Crash

Getting the process to stay alive is necessary but not sufficient. Confirm the configuration is actually healthy:

• Stable VRAM: nvidia-smi should show a plateau during generation, not a slow climb toward the ceiling. A climb indicates cache growth you have not accounted for.
• No swap churn: Watch free -h and vmstat. If you see constant swap activity, the CPU-side layer count is too high for your RAM and --mmap is masking the problem.
• Consistent throughput: Run the same short prompt several times. Wide variance between runs is a sign of paging, not a healthy boundary condition.

The goal is a configuration that is stable under your real workload, not one that merely survives a single launch. If you have been tuning the same model on the same card more than a couple of times, consider writing the final flags into a small launch script so the measured boundary is preserved instead of rediscovered.

Conclusion

llama.cpp’s CUDA out of memory is rarely about the model being wrong and almost always about three interacting levers: layer offload (--n-gpu-layers), the KV cache, and the RAM mapping flags (--mmap, --no-mmap, --mlock). Isolating them — starting conservative, raising layers with nvidia-smi open, adding cache and RAM flags last — turns a cryptic crash into a solved arithmetic problem. For the same kind of memory-budget discipline applied to video generation on consumer GPUs, see our companion piece on Wan2.2-Animate and the offload trade-offs it forces.

How This Article Was Tested

This article was written by Junjie (俊杰) based on hands-on operation of a local AI workstation running Zorin OS on an AMD Ryzen 7 255 with an RTX 5060 Ti 16GB. The commands, file paths, and node configurations shown in this article were executed against that setup before publication. Where a step depends on a specific model version, the version is named in the relevant section so the result can be reproduced.

Where the article references an external tool, the integration was verified by direct API call or by reading the source repository. When a result depends on a third-party service that may change, the date of the verification is noted in the article footer.

What This Article Does Not Cover

Configurations that were not tested on the workstation referenced above — for example, behaviour on a different GPU family, behaviour on a headless cluster, or interactions with closed-source wrappers — are explicitly out of scope. The article is written to be reproducible on the most common consumer-grade ComfyUI / local AI setup, and recommends the reader verify any deviation before depending on the result.

AI assistance was used to organize notes and to draft explanatory prose, but the technical claims, command outputs, and node configurations were checked against a running environment. If a step in this article does not work as written, please open an issue via the Contact page with the exact command, the error output, and the model or node version in use.