If you run ComfyUI on a mid-range GPU, you have almost certainly pasted a launch command that ended in --lowvram or --novram without really knowing what the flag does. The three VRAM management flags in ComfyUI—--lowvram, --novram, and --normalvram—are among the most misunderstood settings in the entire tool, and picking the wrong one either crashes you with torch.cuda.OutOfMemoryError or silently makes your generations two to five times slower.

This guide breaks down what each flag actually does at the memory-management level, how to tell which one you need, and how to verify the change actually took effect. Where I describe internal behavior, this is documentation analysis of ComfyUI’s source and official docs—not fabricated benchmarks. I flag real measurements only where I have them.

The Three Flags and What They Actually Do

ComfyUI exposes three mutually exclusive VRAM modes, controlled by launch flags. You should use exactly one of them, never two.

–normalvram (the default)

With no flag at all, ComfyUI behaves as if --normalvram is set. It loads model weights onto the GPU and keeps them there for the entire session, moving only intermediate tensors and activations as needed. This is the fastest path because weights stay resident and do not need to be re-uploaded between jobs.

You should use this when your largest model (diffusion model plus text encoders and VAE) fits comfortably in available VRAM with headroom for activations. On a 16 GB card running an SDXL-class model, this is usually fine. On the same 16 GB card running a Wan2.2-Animate 14B-class video model, it is almost certainly not.

The failure mode of --normalvram is abrupt: the first job that asks for more VRAM than the card has throws torch.cuda.OutOfMemoryError: CUDA out of memory and the queue halts. If you are hitting OOM only on larger models or batches, you are likely a candidate for a more aggressive offload mode.

–lowvram

--lowvram keeps model weights on the GPU but moves them to regular system RAM between jobs, re-uploading only the layers needed for the current operation. In practice this means most of the model is streamed into VRAM as the compute graph runs, then released. It trades memory for speed: you can run models that are marginally larger than your VRAM, but each generation pays a re-upload cost.

This is the right choice when your model fits in VRAM only some of the time, or when you switch between several large models frequently on a card that cannot hold them all. It is slower than --normalvram but far faster than --novram.

A useful mental model: --lowvram keeps the model “nearby” in shared system memory and moves pieces onto the GPU on demand, whereas --normalvram keeps everything resident.

–novram

--novram keeps model weights entirely on the CPU (in system RAM) and uploads them to the GPU incrementally as each section is computed. It is effectively the most aggressive offload mode. This lets you run models that exceed your VRAM by a large margin—but at the cost of a substantial speed penalty, because the PCIe bus becomes the bottleneck.

Reach for --novram only as a last resort: when the model absolutely will not fit with --lowvram, and you would rather wait than upgrade hardware. The speed difference between --lowvram and --novram is frequently large enough to be the deciding factor between a usable tool and an unusable one.

How to Pick the Right Flag for Your Card

A practical decision sequence that covers most real cases:

1. Model fits in VRAM with headroom → use the default (--normalvram, i.e., no flag). Fastest, simplest.

2. Model fits only sometimes, or you hot-swap large models → add --lowvram.

3. Model exceeds VRAM but you refuse to wait for a cloud GPU → add --novram and accept the slowdown.

4. Still OOM even with --novram → the problem is probably not the diffusion model. Check text encoders, the VAE, and whether you are loading multiple checkpoints at once.

Note that these flags are coarse. Modern ComfyUI and many custom nodes offer finer-grained controls—such as per-node force_offload, or model-specific block-swap and offload-device settings for the Wan video models—that can be more effective than a global --novram. If you are already using those node-level controls, a global VRAM flag may not be necessary at all.

How to Verify the Flag Actually Took Effect

A flag in the right position but attached to the wrong process does nothing. The single most reliable verification is to watch VRAM usage during a generation while monitoring with nvidia-smi:

watch -n 1 nvidia-smi

Observe the MiB used for your process while a job runs. Under --normalvram, usage climbs and stays near its peak. Under --lowvram, you will see memory usage rise and fall as layers stream in and out. Under --novram, usage stays low and the generation is visibly slower.

You can also confirm the active mode from the startup log. ComfyUI prints its memory-management mode (or the relevant offload settings) at launch; check the console output after startup. If you launched via a systemd service or a background script, make sure the flag is actually present in that invocation—many people edit their main.py command in one place and launch from another.

For deeper visibility into what a given model actually needs, run a single job and watch both VRAM (nvidia-smi) and system RAM (free -h). A model that fits under --lowvram will show heavy VRAM utilization with modest RAM growth; a model being pushed to --novram will show near-maximum system RAM usage during generation.

Common Pitfalls and Traps

Using more than one flag. Passing --lowvram --novram is contradictory. Only the last-processed one wins, and what you observe may not match your intent. Use exactly one.

Confusing ComfyUI flags with A1111 flags. Both tools use --lowvram and --novram, but their semantics differ. Do not transplant an A1111 habit onto ComfyUI and assume identical behavior.

Expecting a flag to fix a fundamentally too-small card. A global flag cannot create VRAM. If a 14B video model needs roughly 15 GB and you only have 8 GB, --novram will technically run it but may be pathologically slow—longer than simply renting a cloud GPU. The flag is a knob, not a miracle.

Ignoring node-level offload. For the Wan2.2-family video models specifically, block-swap and offload-device settings often beat a blanket --novram because they keep the hot layers on the GPU while swapping only the cold ones. We covered those exact settings in our guide to running Wan2.2 Animate 14B on 16 GB with BlockSwap.

Chasing VRAM flags before checking which process actually launched. If two ComfyUI instances or a stale daemon are running, your edited flag may never reach the process you interact with. Check ps aux | grep -i comfy to confirm the correct invocation is live.

From a Real Two-Instance Setup

On our own T2 box we run two ComfyUI instances against a single 16 GB card: a systemd-managed instance on port 18188 and a manually launched one on 18189 for Flux 1-dev. That setup makes the flag question concrete—you cannot simply apply one global flag and call it done, because two instances compete for the same VRAM pool.

The takeaway from running both is that VRAM flags are per-process, not global. Each main.py invocation gets its own decision, and the winning strategy is to size each instance’s flag to the heaviest model it will load, not to the card as a whole. If one instance runs an SDXL workflow and the other runs Flux, the Flux instance is the one that needs a more aggressive offload mode. Applying --novram to the SDXL instance would only slow it down for no benefit.

This also means the first diagnostic step when one instance starts OOM-ing is not to tweak flags blindly but to check the other instance’s current VRAM footprint with nvidia-smi. A co-resident process often explains an OOM that your own workflow cannot.

Conclusion

--lowvram and --novram are not magic performance toggles—they are explicit trade-offs between memory footprint and generation speed, applied per-invocation. Use --normalvram (no flag) when your model fits, escalate to --lowvram when it fits only sometimes or you hot-swap models, and reserve --novram for models that genuinely exceed your VRAM. Always verify with nvidia-smi that the flag you think is active actually is, and remember that on multi-instance setups each process needs its own decision sized to its heaviest model.

For the authoritative reference on ComfyUI’s command-line options, see the official ComfyUI documentation and the ComfyUI GitHub repository. If you are specifically wrestling with large video models, the BlockSwap and offload-device approach we documented for Wan2.2 Animate is usually the better lever than any global flag—see our broader open-source video generation stack guide for context.

There are two completely different ways a ComfyUI box can run out of memory, and treating them as the same problem wastes hours. One produces a tidy in-app error. The other silently freezes your desktop, spins your fan to 100%, and then kills the process without so much as a log line. This article is about the second one.

If ComfyUI prints CUDA out of memory and the workflow just stops, that is a VRAM problem — the GPU ran out of dedicated video memory. We have already covered that failure mode in the VAE-decode OOM article. What we are diagnosing here is different: your system RAM (the DDR DIMMs near the CPU) is exhausted, Linux starts swapping to disk, and the kernel’s OOM killer decides ComfyUI is the fattest process on the machine and shoots it. To the user it looks identical to a crash — but the fix is completely different.

The Symptom Pattern That Points at System RAM, Not VRAM

VRAM exhaustion almost always announces itself clearly. You get a torch error mentioning CUDA out of memory, a failed allocation, or a node that reports “Memory” in red. System RAM exhaustion is nastier because it rarely prints anything useful. The telltale signs to watch for:

• The GPU is idle while the machine is unresponsive. Run nvidia-smi in another terminal: if the GPU shows 0% utilization but the box feels frozen, the bottleneck is the CPU/RAM/disk, not the card.
• The process just vanishes. No error dialog, no traceback — the terminal shows Killed or the service reports an exit code of 137 (128 + SIGKILL 9). That is the OOM killer’s signature.
• dmesg shows the smoking gun. Look for a line like Out of memory: Killed process <pid> (python) or oom-kill. This is proof, not guesswork.
• Swap is being hammered. free -h shows the swap column nearly full while the machine crawls.

If you see the first two signs together — idle GPU plus a vanished process — stop chasing VRAM flags and start looking at system memory.

Why ComfyUI Eats So Much System RAM

People assume a GPU-bound tool only cares about VRAM. In practice ComfyUI’s peak system-RAM footprint can run 2–3× the size of the checkpoint you loaded, because of a few compounding effects:

1. Model loading is a RAM staging area. A diffusion checkpoint is typically read from disk into system RAM first, then copied (or progressively streamed) into VRAM. During load, both copies can coexist briefly. A 14 GB video model can transiently need 28 GB+ of CPU RAM just to get onto the card.
2. Custom nodes accumulate without releasing. Python objects referenced from a node’s output cache stay alive until the queue is cleared. A workflow that runs many generations, or one with a misbehaving custom node that never frees its intermediates, will grow the resident set over time.
3. Latent and image buffers live in RAM too. Decoded images, upscaler intermediates, and video-frame latents are often staged in system memory before transfer, adding gigabytes on top of the model.
4. The text encoder and auxiliary models may run on CPU. In --lowvram or hybrid-offload setups, the text encoder can be held in RAM instead of VRAM, and a T5-XXL or umt5 encoder is several GB on its own.

The official ComfyUI troubleshooting documentation groups “system freezing” and “out of memory errors” under the same slow-performance banner and recommends lowering resolution, using the memory-optimization flags, and closing other applications — all of which reduce total pressure but do not directly address a RAM ceiling. See the ComfyUI troubleshooting guide for the baseline advice.

Diagnose Before You Fix: Prove It Is RAM

Run these exact commands during a generation that you suspect triggers the crash. Do not guess — measure.

# 1. Snapshot memory while it runs
free -h

# 2. Watch the highest-RAM processes live
top -o %MEM

# 3. Check whether the kernel killed anything
dmesg -T | grep -iE 'oom|killed process|out of memory' | tail -20

# 4. Confirm the death-by-137 signature on a service
systemctl status comfyui --no-pager

What you are looking for:

• free -h showing available dropping to a few hundred MB while swap fills up.
• dmesg printing an oom-kill entry naming python or python3 as the victim, possibly with a score like oom_score_adj or a memory figure.
• An exit code of 137 — the definitive “killed by SIGKILL” marker.

If dmesg instead shows a CUDA error or the torch traceback, you are back in VRAM territory and the VRAM article applies.

Fix the RAM Ceiling, Not the Symptom

Once you have confirmed system memory is the ceiling, these are the levers that actually help, in rough order of effectiveness:

1. Give ComfyUI a hard memory limit instead of letting it free-run

An unbounded process can balloon until it trips the OOM killer and takes other services with it. On a systemd Linux host, wrap the daemon in a cgroup with an explicit MemoryMax. For example, a unit override that caps a generation daemon at 14 GB:

[Service]
MemoryMax=14G
MemorySwapMax=2G
ManagedOOMSwap=kill
ManagedOOMMemoryPressure=kill

The key detail is ManagedOOMSwap=kill and ManagedOOMMemoryPressure=kill: these let systemd-oomd kill only the cgroup that exceeds its limit, leaving the rest of the machine (your DE, your other services) alive. This is the single highest-leverage change — it converts a system-wide freeze into a contained, diagnosable failure. See the systemd resource-control documentation for the full option set.

2. Restore a small swap as a buffer, but do not rely on it

Two opposite mistakes cause most swap problems. Running with swap disabled means the OOM killer fires the instant RAM is exhausted. Running with a huge swap on a slow disk means the box thrashes for minutes before dying. A few gigabytes of swap on an NVMe gives the kernel room to evict cold pages and lets dmesg log a clean event instead of an abrupt kill. Think of it as a tripwire, not a reservoir.

3. Cut the transient load spike

The load-phase double-allocation mentioned above is often the real culprit — the machine only OOMs for the few seconds a big model is being staged into VRAM. Mitigations:

• Use --lowvram / progressive offload so the model streams in chunks rather than staging the whole file at once.
• Avoid running simultaneous generations on the same host when a large checkpoint is involved; the peak RAM spike of two overlapping loads can be additive.
• Clear the queue and restart ComfyUI between heavy batches so cached intermediates from custom nodes get garbage-collected.

4. Check for a genuine memory leak in a custom node

If resident memory grows monotonically across generations even after the queue is cleared, you may have a leaky custom node. The official custom-node troubleshooting guide recommends --disable-all-custom-nodes followed by a binary search to isolate the offender. This is tedious but is the only reliable way to pin a leak on one node rather than ComfyUI core.

Common Pitfalls That Make It Worse

• Lowering batch_size to “fix” a RAM OOM. Batch size primarily changes VRAM, not system RAM. If dmesg says OOM-killed, shrinking the batch will not save you.
• Adding RAM without checking the actual peak. If the leak is in a custom node, a bigger DIMM just delays the crash. Diagnose the slope of memory growth first.
• Disabling swap entirely “for performance.” This turns a recoverable slow-down into a hard kill. Keep a small, fast swap.
• Confusing --lowvram with --cpu. --cpu runs everything on the CPU and is a last resort; --lowvram only offloads the parts that do not need to stay resident.

Verify the Fix

After applying a change, prove it worked with the same measurements you took during diagnosis:

1. Re-run the workload that previously crashed while watching free -h and top -o %MEM. Peak system RAM should stay below your ceiling.
2. Confirm dmesg no longer logs an oom-kill for the ComfyUI process.
3. If you added a cgroup limit, deliberately exceed it once (e.g. a large batch) and confirm that only the cgroup is killed and the rest of the system stays responsive.

The goal is not “no errors ever” — it is a predictable failure boundary. A box that degrades gracefully and kills only the runaway process is infinitely easier to operate than one that hard-freezes the whole desktop while the GPU idles.

Note on testing: the commands and configuration in this article describe reproducible diagnostic steps and standard Linux mechanisms rather than benchmarked figures. Run them on your own hardware to get numbers specific to your checkpoint and workflow.

You queue a prompt, wait for the sampler to finish, and the saved PNG is pure black. No crash, no red node, no OutOfMemory — just a silent, useless output that looks like it rendered something and then collapsed into zeros. If that describes you, you are hitting one of the most confusing failure modes in ComfyUI: a latent tensor full of NaN (not-a-number) values that the VAE then decodes into an all-black image.

This is a precision problem, not a prompt problem. Your diffusion model isn’t producing a bad image — it’s producing a broken tensor, and ComfyUI is faithfully converting that garbage into black pixels. Below I walk through how to confirm the diagnosis, the three realistic root causes, and the exact launch flags and VAE swaps that resolve each one.

The One-Line Warning That Proves the Diagnosis

Before touching anything, open the terminal that launched ComfyUI and read it. The smoking gun for this failure is a single RuntimeWarning:

RuntimeWarning: invalid value encountered in cast
  img = Image.fromarray(np.clip(i, 0, 255).astype(np.uint8))

That line is the whole story. The sampler produced NaN (or Inf) values in the latent, and when ComfyUI cast those to an 8-bit image, every pixel collapsed to zero. Sometimes the result is gray or has a faint green wash instead of pure black, but the root cause is the same family of problems, and almost all of them are precision-related.

If you see invalid value encountered in cast, the latent was already corrupt before the VAE decoded it. That points the finger at precision in the diffusion model (UNet) or the VAE, which is exactly what the fixes below address. ComfyUI’s own VAE flag help text says the quiet part out loud: the fp16 VAE flag is documented as potentially causing black images. Half precision is the usual culprit.

If you see no warning at all and just a silent black PNG, don’t rule this out — several 2026 reports reproduce exactly that on brand-new model architectures. The diagnostic steps are the same; you just lose the convenience of the one-line confirmation.

Root Cause 1 — The SDXL fp16 VAE NaN Bug

This is the single most common cause for anyone running SDXL or an SDXL-based checkpoint. The original SDXL VAE that Stability shipped numerically overflows in fp16: certain activations exceed the fp16 numeric range, become Inf, then NaN, and you get a black image. It is not your GPU — that specific VAE is simply unstable in half precision.

There are three fixes, in order of preference:

Option A — Swap in the fp16-safe VAE (best)

Download sdxl-vae-fp16-fix, a retrained version of the SDXL VAE that stays inside fp16 range without producing NaNs. Place sdxl_vae_fp16_fix.safetensors into ComfyUI/models/vae/, add a Load VAE node, point it at that file, and wire it into your VAE Decode. This keeps the speed and low VRAM of fp16 while eliminating the black images. The widely used madebyollin/sdxl-vae-fp16-fix on Hugging Face was created specifically to solve this, and community reports are consistent that it works.

Option B — Force the VAE to fp32

Launch ComfyUI with:

python main.py --fp32-vae

This runs the VAE in full precision. No NaNs, slightly more VRAM, and a little slower on decode — but it always works. On a 24GB card the cost is negligible.

Option C — Run the VAE on CPU

If you are VRAM-starved and fp32 VAE tips you into an out-of-memory error, you can offload the decode entirely:

python main.py --cpu-vae

This reliably eliminates the SDXL black-image issue. It is slow — decode can take several seconds instead of a fraction of one — but it is bulletproof when nothing else fits.

Root Cause 2 — The Wrong (or Missing) VAE for Your Model

If there is no Load VAE node in your workflow, ComfyUI uses whatever VAE is baked into the checkpoint. If that VAE is missing or mismatched, you get black or garbage output. The mismatch that bites people most is using an SD 1.5 VAE with an SDXL model, or vice versa. They are not interchangeable.

The fix is to add an explicit Load VAE node and point it at the correct file for your model family:

• SDXL / SDXL checkpoints → sdxl_vae.safetensors (or the fp16-fix version from Root Cause 1)
• SD 1.5 checkpoints → vae-ft-mse-840000-ema-pruned.safetensors
• FLUX → ae.safetensors (never reuse an SD or SDXL VAE)

One more trap worth a quick check: make sure the VAE file actually exists in ComfyUI/models/vae/ and isn’t a zero-byte or truncated download. An empty VAE file decodes to black every time.

Root Cause 3 — The GTX 16-Series fp16 Trap (and New Model Architectures)

This one is hardware-specific. The GTX 1650 / 1660 / 1660 Ti / 1660 Super cards have a known defect in fp16 support — they produce an all-black or all-green image in half precision unless you force full precision. ComfyUI maintainers and a long-running issue thread confirm the behavior: on a 16-series card, fp16 is unreliable across the board.

The fix is a single flag:

python main.py --force-fp32

This sets the UNet, VAE, and text encoder all to fp32. It is the most accurate and slowest path, but on these cards it is the only stable option. Expect noticeably slower generation — the 16-series was never a fast inference card to begin with.

The same pattern shows up on very new architectures before their fp16/bf16 paths stabilize in ComfyUI. The clearest recent example is Z-Image Base, which produced silent black images regardless of precision until an upstream fix landed. If you are running a brand-new model and the Turbo variant works but the Base variant is black, that is a strong signal the problem is upstream and not your setup. In those cases, forcing bf16 on the UNet often helps, because bf16 has a much wider exponent range than fp16 and survives the overflows that kill it — on RTX 30/40/50-series cards bf16 is nearly free.

A Reliable Diagnostic Order

When a black image appears, work through this checklist instead of guessing:

1. Read the terminal. Confirm invalid value encountered in cast or a NaN message. No NaN message → it may still be this problem, but rule out a simple preview/display issue first by opening the saved PNG outside ComfyUI.
2. Identify the model. SDXL → almost certainly Root Cause 1. Brand-new architecture → Root Cause 3. SD 1.5 that worked yesterday → check you didn’t change the VAE or CFG.
3. Identify the card. GTX 16-series → --force-fp32, immediately. Everything else → a precision flag on the VAE first.
4. Apply one flag at a time and restart. ComfyUI reads launch flags at startup, so a flag change means a full restart of the process — refreshing the browser tab does nothing.

A note on choosing the flag: if you have the choice, bf16 is usually the better fix than fp32. bf16 (--bf16-vae, --bf16-unet) keeps the wide numeric range that prevents NaN overflow but uses half the memory and runs faster than fp32. Every RTX 30-, 40-, and 50-series card supports it natively. Reach for fp32 (--force-fp32, --fp32-vae) only when bf16 still misbehaves, or on older cards where bf16 support is weak or absent.

Common Pitfalls That Waste Time

Two mistakes come up again and again in the issue threads:

Assuming it’s the VAE when it’s actually the UNet. If you add the fp16-fix VAE and the image is still black, your problem isn’t the VAE — the UNet is producing NaNs upstream. The fp16-fix VAE only solves VAE overflow; if the latent is already NaN before decode, you need a precision flag on the UNet instead.

Changing the wrong thing and masking the signal. When debugging, change one variable at a time. Start from a known-good official workflow for your model, then reintroduce LoRAs, custom samplers, and precision flags one by one. If a black image appears only after a specific change, that is far more informative than toggling five settings at once.

There’s also a hardware-adjacent tail case worth knowing about. A persistent, non-deterministic black-image regression that survives across clean installs, driver changes, and even different operating systems has occasionally traced back to failing VRAM rather than software. If every software fix fails and the problem worsens at higher resolutions, a VRAM stress test like OCCT can help rule that out before you waste more hours — though this is the rare outcome, not the default.

Verify the Fix and Move On

After applying a precision fix, verify it holds rather than trusting a single lucky run. Generate a small batch (three to five images) at your target resolution. The black-image bug is often intermittent — several reports describe a batch where only some images come out black. A fix that produces clean output across a full batch, at the resolution that was triggering failure, is a real fix.

Your verification checklist is short:

• A clean batch at the previously failing resolution
• No invalid value encountered in cast in the terminal
• The correct VAE and precision flags recorded so you can reproduce the working setup

Once those hold, you have a stable pipeline again. The key to this whole failure mode is remembering that a black image is a precision problem first and foremost — and ComfyUI almost always tells you exactly what went wrong, if you know which warning line to look for.

For more on keeping ComfyUI stable under memory pressure, see our guide on fixing ComfyUI VAE decode out-of-memory errors and the breakdown of attention backends that affect speed and VRAM. The full list of launch flags and their memory behavior lives in the official ComfyUI troubleshooting documentation.

You run ollama run llama3.1:8b, ask a question, and it answers fine. Then a teammate — or a second browser tab, or an agent you wired up — fires a request at the same model, and suddenly the whole thing dies with CUDA out of memory or model requires more system memory than is available. The model worked three seconds ago. Nothing changed. So what actually happened?

The answer is that Ollama began running inference concurrently by default, and every extra concurrent request doesn’t just add a little overhead — it duplicates the KV cache, the single most memory-hungry part of a running model. Once you understand the three environment variables that govern this behavior, the crashes become predictable and preventable.

The Symptom: One Request Works, Two Crash

The classic reproduction looks like this. You have a 16 GB GPU. You load a 13B–14B model quantized to Q4, which leaves you comfortable but not luxurious headroom — say 2–3 GB of free VRAM after the weights and a full context window are loaded. A single request runs at a healthy token rate. Then two requests arrive close together, and the process aborts.

The giveaway is in when the crash happens. It’s not at model load — the model loads fine on its own. It’s precisely when a second concurrent request forces the runner to allocate a second context window. If your logs show a clean start, a first successful generation, and then an OOM abort, concurrency is the first thing to suspect, not your model choice or your quantization.

This is distinct from the more common “my model is too big” OOM, which fails at load time before any token is produced. A concurrency-triggered OOM fails mid-session, after at least one good generation — which is exactly why it feels so confusing.

The Mechanism: Why Parallel Requests Multiply VRAM

When a transformer model generates tokens, its memory footprint splits into two parts. The model weights are loaded once and shared by every request. The KV (key-value) cache, however, holds the attention state for a specific sequence, and it grows with both context length and batch size. Every distinct request needs its own KV cache.

This is the multiplier. If your KV cache for a single 32k context request occupies, say, 2 GB, then two parallel requests need roughly 4 GB, four need 8 GB, and so on. The weights stay the same, but the variable part of your memory usage scales nearly linearly with the number of concurrent requests. On a GPU that was already 90% allocated to weights plus one context, a second context is all it takes to tip you over the edge.

This is also why the same model feels stable in single-user desktop use but explodes the moment you expose it through an API gateway, a local chat UI with multiple sessions, or several automation agents that all happen to call it at once. The model didn’t change — the number of simultaneous sequences did.

Diagnosing the Problem Before It Happens

You can confirm the diagnosis without waiting for a crash. Ollama exposes a /api/ps endpoint (and the ollama ps command) that reports exactly what is loaded and how much memory each model is using. Before you wire Ollama into anything that fires concurrent requests, check the baseline:

ollama ps
# NAME               ID              SIZE      PROCESSOR    UNTIL
# llama3.1:8b        xxxxx           6.1 GB    100% GPU     4 minutes from now

Then start two requests at once (a second terminal with a second ollama run, or two curl calls against /api/generate), and watch the memory climb. On Linux, nvidia-smi in a third terminal shows the exact moment VRAM tops out. If you see the resident usage roughly double and then the process die, you’ve confirmed it.

For headless servers, the two environment variables that matter are set in the environment of the ollama serve process — not in the client that calls it. A very common mistake is setting OLLAMA_NUM_PARALLEL=1 in the shell that runs ollama run; that does nothing, because the constraint lives on the server side, applied per model when requests are admitted.

The Three Environment Variables That Control Concurrency

OLLAMA_NUM_PARALLEL — how many requests run at once

This sets the maximum number of requests a single model will process simultaneously. The default auto-selects either 4 or 1 based on available memory when the model loads. For a single-user desktop, a larger value ages gracefully; for a small GPU serving multiple agents, setting it explicitly to 1 forces requests to serialize instead of stacking multiple KV caches.

# serialize requests to one model on a VRAM-constrained GPU
export OLLAMA_NUM_PARALLEL=1
ollama serve

OLLAMA_MAX_QUEUE — how many requests wait in line

When concurrent requests exceed OLLAMA_NUM_PARALLEL, the excess is not dropped — it is placed in a FIFO queue. OLLAMA_MAX_QUEUE caps how many requests may sit waiting before Ollama rejects the next one outright. The default is generous (512). Lowering it doesn’t save VRAM, but it constrains how deep a backlog of pending, memory-reserving requests can grow, and it lets your callers fail fast with a clear error instead of hanging.

# reject instead of queue above 16 pending requests
export OLLAMA_MAX_QUEUE=16

OLLAMA_KEEP_ALIVE — how long a loaded model lingers

After a request finishes, the model stays resident in memory for a configurable window (default 5 minutes) so the next request doesn’t pay a full reload. In a bursty automation environment, several recently-used models can all sit loaded at once, invisibly consuming VRAM that you think is free. Setting OLLAMA_KEEP_ALIVE to a short value (or 0 to unload immediately) frees that memory, but at the cost of a reload penalty on the next call. It’s a direct tradeoff between latency and headroom.

# unload immediately after idle — reclaim VRAM fast
export OLLAMA_KEEP_ALIVE=0

A fourth variable, OLLAMA_MAX_LOADED_MODELS, deserves a mention. Its default (0) lets Ollama keep up to three models resident per GPU (or three on CPU-only machines). If you’re swapping between a large and small model frequently, that’s convenient; if every megabyte matters, capping it to 1 forces older models to unload before a new one loads.

Common Pitfalls That Make It Worse

• Setting the variables in the client, not the server. The concurrency limits are read by the ollama serve process. If you run Ollama as a system service (systemd on Linux, LaunchAgent on macOS, a Windows service), the environment variables must be injected into that service’s environment — editing the shell that launches ollama run has no effect.
• Confusing KV cache quantization with concurrency. Quantizing the cache (OLLAMA_KV_CACHE_TYPE=q8_0) and enabling flash attention (OLLAMA_FLASH_ATTENTION=1) shrink each cache, which is valuable — but they do not stop the multiplication when parallel requests arrive. Both fixes are worth applying, but they address a different axis of the problem.
• Treating the default NUM_PARALLEL as harmless. The auto-selected default can be 4, meaning four simultaneous KV caches. On a GPU that’s comfortable with one but not four, the crash looks arbitrary because it depends on whether requests overlap in time — which is precisely what makes it intermittent and hard to reproduce.

Verifying the Fix

After setting the variables and restarting the server, reproduce the original failure deliberately. Fire two requests at once and confirm the second is queued (not run in parallel) when NUM_PARALLEL=1 — you should see a single steady generation followed by the queued one, with no memory spike above the single-context baseline in nvidia-smi. Then saturate MAX_QUEUE deliberately and confirm the next request is rejected with a clear error rather than hanging or crashing.

The formal reference is the Ollama FAQ, which documents OLLAMA_NUM_PARALLEL, OLLAMA_MAX_QUEUE, and OLLAMA_MAX_LOADED_MODELS along with their default behavior. For the deeper mechanics of how requests are admitted and queued, the analysis in Rost Glukhov’s write-up on Ollama parallel request handling is a clear, correct reference.

Conclusion

The single most counterintuitive thing about running a local model is that concurrency, not model size, is often what kills it. A model that runs perfectly for one user will OOM the moment a second sequence arrives, because each parallel request clones the KV cache while the weights are shared. The fix is rarely “buy more VRAM” — it’s usually “control how many requests run at once.” Set OLLAMA_NUM_PARALLEL to match your real headroom, cap OLLAMA_MAX_QUEUE so backlog fails fast instead of festering, and be deliberate about OLLAMA_KEEP_ALIVE and OLLAMA_MAX_LOADED_MODELS on multi-agent or automation-heavy boxes. Diagnose with ollama ps and nvidia-smi, reproduce deliberately, and the once-mysterious crash becomes a predictable, documented boundary.

If you’re hitting the more general case of a CUDA out-of-memory on a 16 GB card, see our companion guide on diagnosing llama.cpp CUDA out-of-memory on a 16 GB card.

The Symptom: You Saved VRAM and Prefill Died

You follow the standard advice: quantize your KV cache to free up VRAM on a small GPU. You restart llama-server with --cache-type-k q4_1 --cache-type-v q8_0, watch nvidia-smi confirm the model still fits on the card, and start using it. Generation speed looks fine. But the moment you send a real prompt — a document to summarize, a codebase chunk, a long chat history — it takes minutes before the first token appears, where before it took seconds.

Prompt processing (prefill) that used to run at 1,000+ tokens per second now crawls at 34–100 tokens per second. The longer the prompt, the worse it gets: a 90,000-token input that used to process in under two minutes now blows past the 600-second request timeout entirely. And there is no error message, no red text, no stack trace telling you what went wrong.

This is not your model, not your prompt, and not a driver issue. It is a silent backend fallback inside llama.cpp: when the CUDA flash-attention kernel cannot handle the KV cache types you requested, it quietly moves the entire attention operation to the CPU — and CPU-based attention is tens of times slower.

Diagnosis: Confirm It Is the Fallback, Not the Model

The telltale signature is the asymmetry between prefill and generation. In the report that first isolated this behavior (llama.cpp issue #27109), a Qwen 3.5 hybrid model on an RTX 3090 showed this split:

• q8_0 / q8_0 KV cache: 991–1276 t/s prefill, ~60 t/s generation.
• K=q4_1, V=q8_0: 34–106 t/s prefill, ~60 t/s generation.

Notice what did not change: generation speed stayed identical. Only prefill collapsed. That single observation is your best diagnostic — if per-token decode is unchanged but prompt processing fell off a cliff, you are almost certainly looking at a backend fallback, not a memory-pressure or thermal problem.

Two more confirmation steps, both free:

1. Watch GPU utilization during prefill. Run nvidia-smi dmon in a second terminal while you submit a large prompt with cache_prompt=false. If the card sits near 0% utilization while the prompt is supposedly processing, the work has been moved off the GPU.
2. Flip back to symmetric q8_0. Restart with --cache-type-k q8_0 --cache-type-v q8_0 and resend the same prompt. If speed instantly returns to 1,000+ t/s, the quantized type you picked is the trigger — and you were never constrained by the model at all.

Root Cause: Kernel Selection Fails Silently

The mechanism sits in the CUDA flash-attention kernel selection logic in ggml/src/ggml-cuda/fattn.cu. When llama.cpp tries to run attention with a quantized KV cache, it calls ggml_cuda_fattn_kv_type_supported() to decide whether a compiled kernel can serve that type. In a default build, that check is gated behind a compile-time flag that is OFF:

case GGML_TYPE_Q4_1:
case GGML_TYPE_Q5_0:
case GGML_TYPE_Q5_1:
#ifndef GGML_CUDA_FA_ALL_QUANTS
    return false;  // unsupported in default build
#endif

The default code path also rejects mixed K and V types — a K=q4_0, V=q8_0 combination fails a second guard that requires K->type == V->type. When either guard trips, the kernel selector returns BEST_FATTN_KERNEL_NONE. The graph scheduler then places the whole FLASH_ATTN_EXT operation on the CPU backend.

The critical detail: there is no warning and no offload log message. The scheduler handles it as a normal routing decision, so you get none of the usual “layer offloaded to CPU” output that would tip you off. The first symptom you ever see is the slowdown itself, which is why this bug is so hard to diagnose from the outside.

This is worth distinguishing from the normal dequantization cost. Even on a good build, reading a 4-bit KV cache costs some CPU-time to dequantize back to f16 — but that penalty is a few percent at ordinary context. The collapse documented here is not that. It is the difference between a dequant tax of a few points and an entire 20–30x drop from running attention itself on the CPU.

The Fix: Rebuild With GGML_CUDA_FA_ALL_QUANTS=ON

Because the missing kernels are a compile-time omission, the real fix is to rebuild llama.cpp with the extra kernels enabled. From the llama.cpp source directory:

cmake -S . -B build -DGGML_CUDA_FA_ALL_QUANTS=ON
cmake --build build --config Release -j

This compiles the additional flash-attention quant kernels (q4_1, q5_0, q5_1) and lifts the K == V restriction, so mixed configurations like K=q4_1, V=q8_0 now stay on the GPU. The verification table from issue #27109 is stark — the same RTX 3090, same model, same flag, only the build changed:

The underlying fix also matured into PR #27140, which added dedicated q4_0/q4_1/q5_0/q5_1 → f16 dequantization kernels so that all small KV quant types run within roughly 2% of q8_0 at equal prompt length. If you are on a recent build, check whether that fix has already landed — but do not assume: the guard flag is the only reliable signal.

Common Pitfalls That Mask or Mimic the Fallback

Four traps make this bug worse in practice, because each one sends you looking in the wrong place.

• Assuming the MMQ guard is involved. An older, related issue pointed at a shared-memory guard in the matrix-multiplication path. On many cards that guard passes fine and is not the trigger — issue #27109 confirms it explicitly. Do not pin this on the wrong mechanism.
• Running asymmetric K/V as a “quality boost.” People set --cache-type-k q4_0 --cache-type-v f16 thinking the f16 values buy quality at a small VRAM cost. On a default build this asymmetrical mix silently misses the fast fused path, as documented for AMD HIP in discussion #22411. The VRAM “saving” is real but you also pay the speed penalty — without knowing it.
• Trusting the config instead of measuring. Quantized KV cache can silently fall back to f16 on unsupported architectures. The flag you set is a request, not a guarantee. The only honest check is to load at a fixed context, record VRAM, change the setting, and confirm the number moved.
• Expecting a speedup. Even on a correct build, quantized KV cache is a memory feature, not a speed feature. f16 KV cache is actually faster whenever it fits. Quantize only when f16 genuinely does not fit at the context you need.

Verification: Prove the Fallback Is Gone

After rebuilding, verify rather than assume. Run the same long prompt with cache_prompt=false and compare against your baseline:

1. Record prefill tokens-per-second with the new build and your quantized KV flags.
2. Confirm it is in the same order of magnitude as your q8_0 baseline — not 30x lower.
3. Watch nvidia-smi dmon: the GPU should be pegged during prefill, not idle.
4. Check the server log for offload messages. A clean GPU path prints no CPU-fallback “offloaded layer” lines for attention.

For everyday guidance, the same community trail that diagnosed #27109 lands on a simple ladder: use q8_0 / q8_0 as the safe default on mainline builds. It halves KV cache memory with measurable-but-small quality loss and stays on the fast path everywhere (issue #24485 makes the case for making the extra kernels the default outright). Reach for q4_0 — and mixed K/V — only after rebuilding with the flag, and only when you have measured the VRAM saving actually solves your problem.

Conclusion

If you have already read our guide on fixing slow local LLM chats with KV cache offload and flash attention, this is the deeper trap that guide only gestures at: the same KV cache, when quantized, can silently fall off the GPU entirely. The signature is a prefill collapse with no error and unchanged generation speed. The fix is a rebuild with GGML_CUDA_FA_ALL_QUANTS=ON, verified by watching prefill throughput and GPU utilization return to baseline. Before you conclude your GPU is too small or your model is broken, check whether you are paying for an invisible CPU fallback — it is the cheapest 30x speedup most people will ever find.

You update a custom node, or run ComfyUI’s dependency updater, restart the server — and suddenly half your nodes refuse to load. The console prints a familiar, frustrating block:

A module that was compiled using NumPy 1.x cannot be run in NumPy 2.x as it may crash. To support both 1.x and 2.x versions of NumPy, modules must be compiled with NumPy 2.0. Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

This is not a custom node being “broken,” and it is not fixed by reinstalling ComfyUI. It is a binary ABI incompatibility between NumPy 2.x and a compiled extension (usually PyTorch, OpenCV, insightface, or numba) that was built against NumPy 1.x. This guide walks through exactly what is happening, how to confirm the root cause in under a minute, and the two correct fixes — plus how to stop the clash from returning on the next update.

This article is a documentation-based analysis drawn from the official NumPy troubleshooting guide and confirmed ComfyUI issue reports. No benchmark or “tested on my machine” performance numbers are claimed here; the commands are the same ones maintainers recommend in the linked sources.

What the error actually means

NumPy 2.0 (released June 2024) changed its C extension API. A package that contains compiled C or Cython code — PyTorch, OpenCV, insightface (used by face-swap nodes like ReActor), numba, and many others — is compiled against a specific NumPy ABI version. If that package was built against NumPy 1.x and you later install NumPy 2.x, the compiled module refuses to import, and you get the message above.

The key sentence in the error is the last one before you start Googling: “the easiest solution will be to downgrade to ‘numpy<2’ or try to upgrade the affected module.” That is the entire decision space. You either move NumPy back to 1.x, or move the offended package forward to a NumPy-2–compatible build.

Notice the chain of symptoms it triggers. Because PyTorch itself often fails to initialize NumPy cleanly, the failure can cascade into misleading downstream errors:

• RuntimeError: Numpy is not available on a SaveImage node
• UserWarning: Failed to initialize NumPy: _ARRAY_API not found
• AttributeError: _ARRAY_API not found
• ValueError: numpy.dtype size changed, may indicate binary incompatibility
• Multiple (IMPORT FAILED) summary lines for nodes that were fine yesterday

The common thread is that the first genuine traceback — the one above the IMPORT FAILED summary — points at NumPy, not at the node. According to NumPy’s own troubleshooting guide for ImportError, when you hit this, “search the traceback (from the back) to find the first line that isn’t inside NumPy to see which package has the incompatibility.” That single step is the fastest route from wall-of-text to diagnosis.

Diagnose before you downgrade: use pip check

Do not blindly run pip install "numpy<2" the moment you see the error. There are two distinct failure classes, and they want opposite fixes. It is worth thirty seconds to find out which one you have.

The cleanest confirmation is pip check, which reports incompatible version pins across the whole environment. Run it against the Python interpreter that actually launches ComfyUI:

# Windows portable build (ComfyUI ships its own embedded interpreter)
.\python_embeded\python.exe -m pip check

# Linux / macOS, or any venv-based install — activate the SAME venv ComfyUI uses first
python -m pip check

Representative output when NumPy 2.x is the culprit looks like this:

numba 0.59.1 requires numpy<2.1,>=1.22, but you have numpy 2.3.2.

That one line tells you a compiled package (numba) is pinned to NumPy 1.x while NumPy 2.3.2 is installed. The two options follow directly:

1. Downgrade NumPy below 2 to match the package’s existing pin (fastest, but holds the whole environment at 1.x).
2. Upgrade the offending package to a release built against NumPy 2.x (cleaner long term, but not always available — some packages took months to ship NumPy-2 wheels).

This decision is documented by NumPy maintainers: “The best solution will usually be to upgrade the failing package… however, upgrading may not always be possible because a compatible version does not yet exist.” When the compatible build does not exist, downgrading NumPy is the correct, supported fallback.

Fix 1: pin NumPy back below 2 (the safe range)

The most common fix — and the one most users reach for — is to pin NumPy to a recent 1.x line. There is one important gotcha hidden in this otherwise trivial command: do not pin an ancient release like numpy==1.23.5.

Modern ComfyUI builds ship Python 3.12+, and older NumPy 1.x releases fail to build on those interpreters because current setuptools removed the ImpImporter attribute those old NumPy wheels expected. The NumPy 1.x range that actually supports Python 3.12 is roughly >=1.26. The maintainer-recommended safe range is:

# Windows portable
.\python_embeded\python.exe -m pip install "numpy>=1.26,<2"

# Linux / macOS venv
python -m pip install "numpy>=1.26,<2"

A specific release that appears repeatedly in confirmed ComfyUI issue threads (GitHub issue #5664) is numpy==1.26.4, the last 1.x release. If you prefer a hard pin:

.\python_embeded\python.exe -m pip install numpy==1.26.4

Restart ComfyUI completely afterward — closing the browser tab is not enough. The custom-node imports happen once at server startup, so you must terminate and relaunch the Python process for the change to take effect.

Fix 2: upgrade the NumPy-1.x–compiled package instead

Holding NumPy at 1.x forever is not a strategy; it is a stopgap. The cleaner fix, when your other packages allow it, is to move the offending package to a NumPy-2–compatible build. Most often that package is PyTorch.

If pip check or the traceback points at PyTorch (look for torch/_subclasses/functional_tensor.py or torch/csrc/utils/tensor_numpy.cpp in the warnings), the fix is to upgrade PyTorch to a release compiled against NumPy 2.x. For CUDA builds, install from the CUDA wheel index rather than the default PyPI index:

# Use --index-url, not --extra-index-url, so pip resolves from the CUDA wheel index
.\python_embeded\python.exe -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128

Read the whole traceback before choosing this path. If the incompatibility is coming from opencv-python, insightface, or numba rather than PyTorch, upgrade that specific package (pip install --upgrade opencv-python and so on). The goal in both directions is the same, and it is the sentence NumPy’s guide states plainly: “the goal is one consistent NumPy ABI across the environment.”

Why it keeps coming back — and how to stop it

This error rarely appears out of nowhere. It almost always arrives right after one of two events, and understanding which one is what prevents the recurrence:

• You ran ComfyUI’s “update all” or a dependency updater. Scripts like update_comfyui_and_python_dependencies.bat on Windows Portable reinstall the full requirements.txt and can pull in a newer NumPy while other packages still pin 1.x. In GitHub issue #9156, a user triggered exactly this by running the updater, which silently bumped NumPy to 2.3.2 and broke most custom nodes.
• You installed a new node pack. A single node’s requirements.txt can pip install a NumPy 2.x (or, less commonly, force a NumPy 1.x downgrade under other nodes), shifting the ABI underneath everything else.

Two habits prevent most of this. First, preview before you install — pass --dry-run to see what a node’s requirements will actually change:

.\python_embeded\python.exe -m pip install -r custom_nodes\SomeNode\requirements.txt --dry-run

Second, prefer ComfyUI-Manager for node installs and updates rather than raw pip or the manual dependency updater. Manager installs dependencies into the correct embedded interpreter and is more conservative about bumping shared core packages like NumPy and PyTorch. Maintainers and long-time users in the issue threads repeatedly steer people back to Manager for exactly this reason: it is the least likely path to clobber the environment.

If you must update PyTorch or CUDA manually, do it deliberately and check pip check immediately afterward. Treat a clean pip check as the exit condition for any dependency change, the same way you treat a passing test suite as the exit condition for a code change.

Verification: how you know it is actually fixed

Pinning NumPy and restarting is only half the job. Confirm the fix before you queue a real generation:

1. Restart ComfyUI from a visible terminal (or open the desktop log) and confirm the raw import error is gone, not just buried. The node summary lines should read 0.0 seconds without (IMPORT FAILED).
2. Run pip check again and confirm it exits with no incompatible pins.
3. Queue the smallest known-good workflow using one of the previously failing nodes, and confirm it executes.
4. Watch for the secondary symptoms — if RuntimeError: Numpy is not available or Failed to initialize NumPy reappears on a SaveImage node, the ABI clash is not fully resolved and you still have a mixed environment.

For more context on reading ComfyUI import failures generally, see our guide on diagnosing “No module named” errors and broken custom nodes, and for the wider symptom-first workflow, the official ComfyUI troubleshooting guide is the authoritative reference. The definitive write-up on this specific ABI error lives in the NumPy ImportError troubleshooting documentation.

Conclusion

The “compiled using NumPy 1.x” error is intimidating because it says very little and sprays misleading secondary warnings. But it resolves to a binary choice that NumPy’s own documentation spells out: either downgrade NumPy below 2, or upgrade the compiled package to a NumPy-2 build. Diagnose with pip check first, pin to the safe numpy>=1.26,<2 range if you downgrade (never an ancient 1.23.x), and use --dry-run plus ComfyUI-Manager so the next update doesn’t silently recreate the clash. One consistent NumPy ABI is the whole fix.

The Problem: A Model That Shouldn’t Fit

Qwen3-235B-A22B sounds like it needs a rack of H100s. Read the model card and you’ll see 235 billion total parameters — on paper that’s a ~130 GB GGUF file in Q4_K_M quantization, nowhere near a 16 GB consumer card. Yet people run it at usable speeds on a single RTX 4060 Ti or 3090.

The reason is in the fine print: it’s a mixture-of-experts (MoE) model, not a dense one. Of those 235B parameters, only 22 billion are activated per token. The model has 94 layers and 128 “expert” feed-forward networks per layer, but the router only consults 8 of them for any given token. That means ~90% of the weights sit idle on any forward pass.

llama.cpp exposes two flags — --n-cpu-moe and --override-tensor — that let you exploit this sparsity directly. Instead of the crude all-or-nothing layer offloading that --n-gpu-layers gives you, these flags let you strand the expert tensors in system RAM while keeping the small “always active” parts of the model on the GPU. This article is a documentation-based walkthrough of what those flags actually do, where the common guides get them wrong, and how to verify your own configuration. No benchmarks are fabricated here — everything below is drawn from the llama.cpp source, its CLI reference, and the official Hugging Face model cards, and is reproducible on your own hardware.

First, Understand the MoE Memory Layout

Before touching a flag, you need to understand why offloading experts is different from offloading layers. In a dense model like Llama-3.3-70B, every one of the 70 billion parameters participates in every token. Offload any of it to CPU RAM and you slow down every single token — there is no “cheap” weight to strand.

An MoE model splits each transformer layer into two kinds of weights:

• Always-active parameters — the attention module, the embedding, the router, and a small “shared expert” FFN that fires on every token. These are relatively tiny (a few percent of total parameters) but they’re touched on every token, so they want to live on the GPU.
• Routed expert FFNs — the bulk of the parameters. For Qwen3-235B-A22B, each layer has 128 experts but only 8 activate per token. These tensors are enormous individually but each one is used rarely. This is the part you can afford to keep on slower CPU RAM.

In the GGUF tensor naming, the routed experts show up per layer as blk.N.ffn_gate_exps.weight, blk.N.ffn_up_exps.weight, and blk.N.ffn_down_exps.weight. The shared expert is blk.N.ffn_shexp.weight. This naming is the key to understanding how the offload flags work.

Flag #1: --n-cpu-moe N

The official CLI reference describes --n-cpu-moe N (short -ncmoe) as: “keep the Mixture of Experts (MoE) weights of the first N layers in the CPU.” Its sibling --cpu-moe (-cmoe) keeps all MoE weights on CPU.

Two details here matter and are frequently misreported:

1. It counts layers, not experts. Some community guides describe --n-cpu-moe 20 as “offload 20 of the 28 experts,” which is backwards. The flag moves the expert FFN weights of the first N layers to CPU. The attention, router, and shared expert of those layers stay wherever --n-gpu-layers put them.
2. It is a blunt per-layer cutoff, like --n-gpu-layers but specialized to expert tensors. You can’t use it to keep, say, only the attention on CPU while experts of the same layer stay on GPU — the whole layer’s expert block moves together.

Here’s a minimal launch that loads everything possible to the GPU but strands the expert FFNs of the first 40 layers on CPU:

./llama-server -m Qwen3-235B-A22B-Q4_K_M.gguf \
  --n-gpu-layers 99 \
  --n-cpu-moe 40 \
  -c 8192

--n-gpu-layers 99 (or -ngl 999) requests “push everything possible to the GPU.” --n-cpu-moe 40 then overrides that for the expert tensors of the first 40 layers, sending them to system RAM instead. The result: attention, router, and shared experts stay on GPU where they’re cheap to compute, while the huge rarely-touched expert matrices live in RAM.

The trick barely helps a dense model — there’s no sparse expert tensor to strand — so check the model card first. If it doesn’t say “mixture-of-experts,” --n-cpu-moe won’t save you.

Flag #2: --override-tensor (surgical control)

If --n-cpu-moe is a blunt layer cutoff, --override-tensor (short -ot) is a scalpel. It takes a regex that matches individual tensor names and assigns them to a specific buffer type. The syntax is pattern=buffer, with multiple patterns joined by commas.

The original PR that added it (llama.cpp #11397) offers the canonical MoE example:

./llama-server -m Qwen3-235B-A22B-Q4_K_M.gguf \
  --n-gpu-layers 99 \
  -ot "exps=CPU"

The pattern exps matches every routed-expert tensor (ffn_gate_exps, ffn_up_exps, ffn_down_exps) and sends them to CPU, while the shared expert (ffn_shexp) and everything else stays on GPU. That’s the cheapest way to reclaim VRAM, because the routed experts are the bulk of the model and the most rarely activated.

For finer control you can target specific layers. To offload only layers 40–47’s expert FFNs (the “late layers” that activate least often) while keeping early-layer experts on GPU:

-ot "blk\.(4[0-7])\.ffn_(gate|up|down)_exps\.weight=CPU"

Two gotchas that trip people up:

• Multiple -ot flags are deprecated. Passing -ot twice can silently send all matched experts to the last target. Combine everything into a single comma-separated regex instead.
• The pattern is a regex, and it’s easy to typo into matching nothing. Launch once with -ot ".*=CPU" and the -v (verbose) flag to print every tensor name being overridden, then build your real pattern from the actual names.

The Misinformation You Should Ignore

Because this is a fast-moving area, a few widely-circulated guides now contradict the current llama.cpp documentation. Two claims in particular deserve correction:

1. “You must add --split-mode row for --n-cpu-moe to work.” The current docs/multi-gpu.md marks row as deprecated, superseded by tensor. And critically, --split-mode tensor is not implemented for MoE architectures — the docs list Grok, DeepSeek2, Nemotron-H, and other MoE/hybrid families as explicitly unsupported, and attempting it fails with LLAMA_SPLIT_MODE_TENSOR not implemented for architecture '...'. On a single GPU the whole question is moot: --split-mode governs multi-GPU splitting. For one 16 GB card, don’t pass it at all.

2. Conflating “experts” with “layers” in the --n-cpu-moe argument. The flag counts layers. Treating it as an expert count produces a configuration that’s off by a factor, and authors then publish numbers that don’t reproduce.

The safe rule: for a single-GPU setup, prefer --n-cpu-moe for a quick fit and -ot for a deliberate one. Leave --split-mode alone.

Diagnostic Steps: Get It Right the First Time

Offloading is a memory-budget problem, and the fix is to measure rather than guess. Work through this order:

1. Record the baseline. Run nvidia-smi before launching and note free VRAM. Subtract a few hundred MiB for driver overhead.
2. Start conservative. Load with --n-gpu-layers 99 --cpu-moe (all experts on CPU) and a modest -c 4096. Confirm the server boots and produces tokens. This is the guaranteed-fit floor.
3. Reclaim experts in layers. Replace --cpu-moe with a smaller --n-cpu-moe N, or move from a blanket exps=CPU to a per-layer -ot pattern that only offloads the late layers. Watch nvidia-smi after each change and stop when used VRAM settles near ~80% of capacity, leaving headroom for the KV cache and activations.
4. Add context last. Only after the weight split is stable should you widen -c. The KV cache is the classic hidden variable — it grows with context length and can blow the budget even when the weights fit.

Remember the quantization math too: a Q8_0 or F16 file is two to four times larger than its Q4_K_M sibling. The same offload settings that work for Q4_K_M will OOM on Q8_0, so know your file’s footprint before choosing values.

Common Pitfalls and How to Verify

Several failure signatures have specific causes:

• OOM at startup or during prefill. Too many expert layers still on GPU, or too large a context. Lower -c first, or increase --n-cpu-moe.
• Decode barely faster than full CPU. Your -ot regex matched too much — likely attention or shared-expert tensors rather than only exps. Re-verify with -v that it hits only the routed experts.
• Runs, but VRAM slowly climbs toward the ceiling. The KV cache is growing across a long conversation. Size it explicitly or cap context.

To confirm a healthy configuration, look for three things:

1. Stable VRAM. nvidia-smi should show a plateau during generation, not a steady climb.
2. No swap churn. Watch free -h and vmstat; constant swap means the CPU-resident expert count is too high for your RAM.
3. Consistent throughput. Run the same short prompt several times — wide variance between runs signals paging, not a healthy boundary.

Also know the honest limit. Routing experts to CPU trades VRAM for memory bandwidth, so fast dual-channel DDR5 matters more here than anywhere else in a local-AI build. Slow single-channel memory, or integrated graphics borrowing system RAM, can erase the gain entirely. This isn’t a free lunch — it’s you spending RAM bandwidth to buy back VRAM headroom.

Conclusion

Running a 235-billion-parameter model on a 16 GB card isn’t magic; it’s a property of MoE sparsity. Because only 22B of those parameters activate per token, llama.cpp lets you strand the bulky routed-expert weights in system RAM with --n-cpu-moe (a per-layer cutoff) or --override-tensor (a regex scalpel), while the small always-active attention and shared-expert tensors stay on the GPU.

The discipline is the same as any VRAM problem: match your offload to a measured budget, not a guess. Start conservative with all experts on CPU, reclaim them layer by layer while watching nvidia-smi, and add context last. Ignore the stale advice to pass --split-mode row — on a single GPU it’s irrelevant, and it’s deprecated anyway. Get the layer-vs-expert distinction right, confirm your -ot pattern with -v, and a model that once seemed to require a workstation will run on the GPU you already own.

If you’re still running into CUDA out of memory with dense models rather than MoE, see our companion guide to --n-gpu-layers, --mmap, and layer offloading, which covers the dense-model case these flags don’t help with.

Why a 429 Is Different From Every Other Error

A 429 is the provider telling you your credentials are fine and your request was refused on volume grounds — not a malformed request, not a bad key. That matters because the fix is time and pacing, not a code change. OpenAI’s documentation is explicit that a Retry-After header “does not mean that quota, billing, or other errors that require user action can be resolved by retrying.” In other words, there are several distinct 429s hiding behind one status code, and treating them all the same is how you burn your retry budget on nothing.

The error splits cleanly into two families:

• 4xx client errors (other than 429 and 408) — your fault. Bad JSON, a too-long context window, an expired key. Retrying is pointless; fix the request.
• 429 and 5xx errors — the provider’s constraint. These are retryable, but how you retry depends on which flavor of 429 you actually hit.

Within the 429 family itself there are at least four situations worth distinguishing, because the correct reaction differs:

• Per-minute request or token cap — you sent too fast. Waiting for the documented window fixes it.
• Spend or credit cap — the account is out of money or hit an org/project spend limit. Waiting does not fix it; retrying burns quota on nothing.
• Acceleration / ramp limit — your traffic spiked faster than the provider allows, even under your nominal caps. The fix is a gradual ramp, not a retry storm.
• Upstream gateway throttle — an aggregator relaying a provider’s own limit. Retrying the same route makes it worse; fail over instead.

OpenAI distinguishes the spend-cap case with error codes like credit_balance_exhausted or organization_spend_limit_exceeded, while Anthropic surfaces an enforced_spend_limit_reached detail code with no retry-after header. Learn to read the error body, not just the status line.

Read the Response Headers Before You Retry

The single most useful thing a 429 response gives you is in its headers, and most people skip it. When a retry-after header is present, it tells you exactly how many seconds to wait before the next attempt is likely to succeed. Retrying sooner just gets you another 429.

Each provider structures this differently, which is worth internalizing if you use more than one:

Anthropic’s token-bucket model is a subtle but important detail: capacity replenishes continuously rather than resetting on a fixed clock boundary. That is why a burst can trip a 429 seconds after a previous burst succeeded, and why the reset value is a timestamp, not “wait until the top of the minute.” Anthropic is also explicit that “a rate of 60 requests per minute might be enforced as 1 request per second” — meaning short bursts can exceed a limit that a naive per-minute counter thinks you’re under.

Here is the key resilience rule: a header set belongs to whichever model actually served the request, not the endpoint you called. If you route through an OpenAI-compatible aggregator, some routes return a full x-ratelimit-* set and others return none at all. Your parser must degrade gracefully to plain backoff when a header is missing, rather than throwing a KeyError.

Implement Exponential Backoff With Jitter (Not Fixed Sleeps)

The reason fixed delays fail is the thundering herd problem. Imagine a thousand clients all hit a limit at the same instant. If every one of them waits exactly one second and retries, they all collide again, get throttled again, and repeat the cycle in lockstep forever. Exponential backoff spaces those retries over a growing interval; jitter adds randomness so clients stop synchronizing.

Here is a minimal, correct retry loop that honors retry-after when present and falls back to jittered backoff when it is not:

import random
import time

import httpx

RETRYABLE = (429, 500, 502, 503, 504)

def call_with_backoff(url: str, payload: dict, headers: dict,
                      max_retries: int = 5) -> httpx.Response:
    for attempt in range(max_retries):
        resp = httpx.post(url, json=payload, headers=headers)
        if resp.status_code not in RETRYABLE:
            return resp

        # Prefer the server's explicit instruction.
        retry_after = resp.headers.get("retry-after")
        if retry_after:
            try:
                wait = float(retry_after)
                time.sleep(min(wait, 120.0))
                continue
            except ValueError:
                pass  # fall through to backoff

        # No usable header -> jittered exponential backoff.
        base = min(2 ** attempt, 16.0)   # 1, 2, 4, 8, 16s cap
        wait = random.uniform(0, base)   # full jitter
        time.sleep(wait)

    raise RuntimeError("max retries exhausted")

Three details in that loop matter more than they look:

• Cap the sleep. Most providers will not make you wait minutes for a per-minute cap, but a rogue or malformed retry-after value should never stall your worker for an hour. Clamp it.
• Full jitter beats “equal jitter.” random.uniform(0, base) scatters clients across the entire window, which breaks synchronization better than a small random bump around a fixed value.
• Do not stack a second retry layer on top of the SDK’s. The official OpenAI and Anthropic clients already retry transient failures a couple of times and already honor retry-after. If you wrap every call in your own loop and leave the SDK retries on, your “3 attempts” silently become nine, and the waits compound. Either own the loop yourself, or raise the SDK’s max_retries and let it manage the pacing — not both.

Distinguish What “Waiting” Will and Won’t Fix

The most common production bug around 429s is a retry loop hammering an exhausted credit balance. The request fails, you wait ten seconds, it fails again, you wait twenty seconds — and you loop until your alerting notices, never actually recovering. The signal is in the body, not the status line.

A quick decision tree to run on every 429:

• Is there a retry-after header? If yes, honor it, capped. If the error body names a spend/credit cap, skip straight to “fix billing.”
• Does the error body name billing? Look for credit_balance_exhausted, organization_spend_limit_exceeded, project_spend_limit_exceeded, or Anthropic’s enforced_spend_limit_reached. Do not retry — log it, alert a human, and stop.
• Did traffic just spike? If you ramped requests faster than your baseline, back off to your previous rate and grow it gradually (OpenAI suggests no more than ~50% increase every 15 minutes once near the TPM ceiling).
• Is the limit the provider’s, not yours? Anthropic’s 529 overloaded_error and a provider-side 503 mean their servers are at capacity, not that you misbehaved. Back off calmly, or fail over — retrying the same route harder makes it worse.

Keeping those four cases separate is what separates a graceful degradation from a cascading failure.

Build a Fallback Chain for When Retries Aren’t Enough

Retrying the same provider only helps for transient limits. For extended outages or a hard spike, you need somewhere else to send the request. A fallback chain routes to a secondary model or provider once the local retry budget is spent, and it is the single highest-leverage reliability change you can make to an LLM-backed service.

The elegant part: if you route through an OpenAI-compatible endpoint, failing over is often just a model string change, not a second integration. Here is a sketch that tries a primary model, then a cheap fallback, then gives up cleanly:

ROUTES = [
    {"url": "https://api.openai.com/v1/chat/completions",
     "model": "gpt-4o", "key": "PRIMARY"},
    {"url": "https://api.anthropic.com/v1/messages",
     "model": "claude-sonnet-4-5", "key": "FALLBACK"},
]

def call_with_fallback(payload: dict) -> dict:
    last_err = None
    for route in ROUTES:
        try:
            resp = call_with_backoff(route["url"], payload,
                                     auth_headers(route["key"]),
                                     max_retries=3)
            if resp.status_code == 200:
                return resp.json()
            last_err = f"{route['model']} -> {resp.status_code}"
        except RuntimeError as e:
            last_err = str(e)
            continue  # retries exhausted on this route; move on
    raise RuntimeError(f"all routes failed: {last_err}")

Two design notes worth calling out:

• Bound each route’s retries. Don’t let a single provider eat the entire global retry budget while a healthy fallback sits idle. Give each route a small max_retries and move on.
• A circuit breaker beats retrying harder. Once a route’s error rate crosses a threshold (say 50% over 60 seconds), open the circuit and route around it entirely for a cooldown period, letting a few probe requests through afterward. This stops you from piling latency onto a provider that is already down and gives it room to recover.

Verify Your Fix Under Load

The mistake people make after wiring this up is never testing it under the failure mode it was built for. A retry loop that looks correct in review can still synchronize every worker, exhaust a thread pool, or loop forever on a spend-cap 429. A few concrete checks before you call it done:

• Simulate a 429. Point your client at a stub that returns 429 with a retry-after: 2 header and confirm it waits (roughly) two seconds and succeeds on retry, rather than spinning.
• Simulate a headerless 429. Confirm it falls back to jittered exponential backoff and stops after max_retries, rather than looping forever.
• Simulate a billing 429. Return a 429 whose body names credit_balance_exhausted and confirm the client does not retry — it logs and aborts.
• Watch the logs. Track status codes, latency, token counts, and retry counts per route. A 429 rate above ~5 per minute, or fallback usage above ~50%, is your early warning that a primary provider is in trouble and you should investigate before it becomes an outage.

A small but revealing final detail: spend five minutes checking whether your SDK already retries for you. The installed OpenAI client defaults to two retries, honors retry-after-ms first and then retry-after, caps a server-directed delay around two minutes, and refuses to retry when the server asks for longer than that. If a 429 is surfacing in your own code, it has often already failed three times inside the SDK — which is a hint that the limit itself, not your retry logic, is the problem worth fixing.

Conclusion

429 rate-limit errors are not anomalies; they are the environment telling you how fast you are allowed to go. The reliable response is a layered one: read and honor retry-after, fall back to jittered exponential backoff when it is absent, refuse to retry billing-cap 429s, and route around a struggling provider with a fallback chain. None of this is glamorous, but it is the difference between a service that degrades gracefully under load and one that takes a short throttle and amplifies it into a full outage.

For more on the operations side of running local and hosted AI reliably, see our guide on ComfyUI automation via WebSocket events and history polling and our post on building a Python harness for the ComfyUI API.

Primary References

• OpenAI API error codes — rate limits and retry guidance
• Anthropic rate limits — headers and spend caps