Category: Local AI Operations

If you have an RTX 5090, 5080, 5070 Ti, or 5070 and tried to speed up ComfyUI with SageAttention, you probably hit a wall before anything actually ran. The install either fails outright with a Triton compilation failed error, or the build finishes and ComfyUI silently falls back to a slower attention path. Neither is your fault, and neither is a mystery once you understand what changes when CUDA targets Blackwell (compute capability sm_120).

This guide is a documentation analysis of the official SageAttention repository and the ComfyUI startup-flag docs, cross-referenced against the issues real RTX 50 users have filed. It is not a fabricated benchmark report — I have no RTX 50 card on hand to run one. What follows is the reproducible procedure and the reasoning behind each step, so you can tell whether SageAttention is actually doing anything on your own machine.

Why SageAttention Breaks on Blackwell

SageAttention is a quantized attention kernel. Instead of computing attention in fp16/fp32 like FlashAttention, it quantizes the K and V matrices to 8-bit and does the bulk of the multiply in int8, then de-quantizes the result. The paper reports a 2–5× speedup over FlashAttention without measurable accuracy loss, and on an RTX 5090 the maintainers measured roughly 2.7× FlashAttention2 throughput (SageAttention repo). The catch: quantization kernels are written directly against specific NVIDIA architectures, so a kernel compiled for one GPU family does not run on another.

Older cards (Ampere sm_80/sm_86, Ada sm_89, Hopper sm_90) had been covered in SageAttention’s build for a while. Blackwell landed with sm_120, and for a long window the packaged build simply did not include a code path for it. Users trying to install on an RTX 50 card would either get a CUDA “unsupported architecture” error during compilation, or a cryptic Triton compilation failed from the kernel loader because the generated code did not match the running GPU.

The fix the maintainers shipped (issue #107, “Compile SageAttention CUDA 12.8, TORCH 2.7.0 – Blackwell”) is exactly what the issue title implies: add arch=compute_120,code=sm_120 to the NVCC flags so the CUDA extension compiles a real Blackwell target instead of guessing. If you install a prebuilt wheel from an older pin, you will not get this target, which is why “pip install a wheel” tutorials keep failing for 50-series owners.

Symptom: The Three Ways a Broken SageAttention Install Shows Up

Before you spend an evening recompiling, confirm that SageAttention is actually your bottleneck and not some unrelated CUDA/torch mismatch. The failures cluster into three distinct shapes:

• Build-time failure. pip install or python setup.py install errors out with No kernel for device was found, an unsupported gpu architecture NVCC message, or Triton compilation failed. This is the classic missing-sm_120 symptom and means SageAttention never even got installed.
• Silent fallback. The install “succeeded” but ComfyUI’s startup log does not report loading SageAttention, and generation speed is unchanged. ComfyUI does not always warn when a requested backend is unavailable — it can simply use its default (usually SDPA or sub-quadratic) and move on.
• Version skew. SageAttention compiled against one torch/CUDA combo but imported under another. Because the extension is a compiled CUDA binary loaded into torch’s address space, a mismatch here typically surfaces as a segfault or an undefined symbol at import time, not a clean Python error.

Step zero in every case: read the actual terminal output, not the ComfyUI web UI. The backend selection and any kernel-load errors are printed to the console where you launched python main.py. The web UI almost never surfaces them.

How to Install SageAttention for Blackwell Correctly

The reliable path is to build from source against the exact torch and CUDA versions ComfyUI is already using, rather than pulling a wheel that matched someone else’s environment.

First, pin your dependencies before you compile. Check what torch reports and what CUDA toolkit your driver exposes, then make sure they agree with what you are about to build against. A common failure is having CUDA 12.8 toolkit but a torch wheel built for 12.4 — SageAttention will compile against one and load against another.

The build needs the Blackwell target forced into the flags. In the SageAttention extension setup, the line that matters is the NVCC target list. In older checkouts it enumerated older architectures only; the fix adds the compute_120 entry explicitly:

NVCC_FLAGS += ["-gencode", "arch=compute_120,code=sm_120"]

If you are on a current checkout, this is already present. If you are on an old pin, that is precisely why it fails — update to the current SageAttention rather than patching an outdated snapshot by hand. The maintainers also note that SageAttention3 kernels for Blackwell live under a dedicated sageattention3_blackwell directory in the repo, so if you want the newest path, that is where to look (SageAttention3 paper).

A note on Triton

Several RTX 50 issues ultimately trace to Triton, not SageAttention itself. If your Triton build predates Blackwell support, its generated kernels will also fail to find a valid device target. The practical advice from the issue threads is to install a Triton release that declares Blackwell support (a recent 3.x), and to not mix a hand-built Triton with a wheel-built SageAttention and vice versa. When the two disagree on CUDA runtime version, you get the same opaque Triton compilation failed message for a completely different underlying reason.

Enabling It in ComfyUI (And Verifying It Took)

Having the library installed does nothing until ComfyUI is told to use it. The attention backend is selected at launch with a flag. Running python main.py --help in your ComfyUI directory lists everything, and the startup-flag docs confirm --use-sage-attention as the switch (see ComfyUI Startup Flags). Launch with:

python main.py --use-sage-attention

Two caveats matter here. First, the attention flags are mutually exclusive — the cross-attention method flags (--use-split-cross-attention, --use-quad-cross-attention, --use-sage-attention, etc.) pick one backend, and passing several just means the last one wins or produces undefined behavior. Second, the docs mark the SageAttention flag as intended for testing/advanced use, and note that enabling it disables attention upcasting — which is fine for speed but is why some workflows that look identical produce slightly different pixels.

Verification is the part people skip. Do not trust that a clean launch means SageAttention is active. After ComfyUI starts, confirm it in the console: look for the line that reports which attention backend loaded, then run a single generation and confirm it completed without a runtime CUDA error. A common trap is that the flag silently has no effect because the library was installed into a different Python environment than the one running main.py — ComfyUI portable installs on Windows, for example, use their own embedded Python, so a pip install into your system Python is invisible to it. Install into the same interpreter that launches ComfyUI.

A quick sanity check

Inside that Python environment, import the module and confirm it resolves:

python -c "import sageattention; print(sageattention.__file__)"

If that import succeeds from the same interpreter ComfyUI uses, and your launch command carries --use-sage-attention, and the console log reports SageAttention as the active backend, then it is genuinely on. If any one of those three is false, you are looking at a silent fallback and have not actually enabled it.

Common Pitfalls to Avoid

• Wheel from the wrong arch. A wheel built for Ada/Hopper will install cleanly and then produce zero-acceleration or a runtime error on Blackwell because the GPU target is absent. Build from source or use a Blackwell-targeted wheel.
• Environment mismatch. Installing into system Python while ComfyUI runs from a portable/venv Python is the single most common “I enabled the flag and nothing happened” cause.
• Assuming speed means correctness. SageAttention trades a tiny amount of numerical precision for speed. For most image/video generation this is visually irrelevant, but if you are chasing pixel-exact reproducibility you should A/B against the default backend before adopting it wholesale.
• Skipping the log. ComfyUI’s web UI will not tell you which attention kernel loaded. The console is the only source of truth for backend selection.
• Treating SageAttention as the root cause. On a 50-series card, several distinct failures (cuDNN frontend, torch/CUDA skew, Triton) all overlap with “attention-related” symptoms. If SageAttention is already correctly enabled and still slow, the problem is likely elsewhere — see the related coverage on attention backend selection in ComfyUI Attention Backends Explained and on Blackwell-specific cuDNN errors in ComfyUI cuDNN Frontend Error: No Valid Execution Plans Built.

Conclusion

SageAttention on an RTX 50-series GPU fails for one dominant, fixable reason: the build lacked a sm_120 Blackwell target, which surfaces as a Triton compilation failed error or a silent performance no-op. The solution is to build (or obtain) a Blackwell-targeted SageAttention against the exact torch/CUDA pair ComfyUI uses, install it into the same Python environment that launches ComfyUI, enable it with --use-sage-attention, and then — crucially — verify the backend actually loaded from the console log before believing the speedup. Everything else — environment mismatches, mutually-exclusive attention flags, skipped verification — is a pitfall that makes the real fix look like it did not work. Get these three variables aligned and the install stops being a mystery.

The Symptom: 100% CPU, GPU Sitting Idle

You download a quantized Wan 2.2 text encoder — something like umt5-xxl-encoder-Q3_K_S.gguf — wire it into your workflow through a CLIPLoader (GGUF) node, hit Queue, and watch your CPU peg at 100% while the taskbar shows your GPU twiddling its thumbs. Encoding a single prompt takes what feels like an eternity, and you’re left wondering whether you bought the wrong hardware.

The instinctive fix, shared all over Reddit and Discord, is to “just force the CLIP loader onto the GPU.” On paper it makes sense: the GPU is idle, the CPU is choking, so move the work across. But before you reach for a SelectCLIPDevice node or a MultiGPU wrapper, you need to understand why that node routes to the CPU in the first place — because on a single-GPU machine, forcing it to cuda:0 is often the quickest way to trade a slow-but-working render for an outright out-of-memory crash.

Why the Text Encoder Runs Once — and Why That Matters

Text encoding in ComfyUI is not a per-step operation. Unlike the diffusion UNET, which runs its forward pass once for every single sampling step (anywhere from 20 to 60+ times per generation), the text encoder runs exactly once per prompt, at the very start of the pipeline. It converts your tokens into conditioning embeddings, and those embeddings are then reused for every subsequent step.

This is the foundation of the standard advice that “text encoding is fine on the CPU, it barely costs you anything.” For the small CLIP encoders used by SD1.5 and SDXL, that advice is perfectly correct: a ~120 MB CLIP-L encoder can churn through a prompt in a fraction of a second on any modern CPU, and keeping it off the GPU freees precious VRAM for the model that actually dominates runtime — the UNET.

But that reasoning silently assumes a small encoder. With Wan 2.2 and the umt5-xxl family, it completely falls apart, and this is the root of the confusion you’re running into.

The Real Cause: umt5-xxl Is a Full LLM, Not a Tiny CLIP

The Wan family does not use a dinky CLIP text encoder. It uses umt5-xxl, a massive encoder-decoder language model whose encoder alone is roughly 11.4 GB at bfloat16. Even after aggressive quantization, it stays large: the official fp8 build (umt5-xxl-enc-fp8_e4m3fn.safetensors) is about 6.7 GB, and a GGUF Q3_K_S quant still lands in the 4–5 GB range. For reference, that single text encoder is larger than some entire SD1.5 checkpoints.

So when you load a quantized umt5 through CLIPLoader (GGUF) and it lands on the CPU by default, you’re asking a consumer CPU to perform a full forward pass through an 11-billion-parameter-class model. Of course it pegs at 100% and takes a while. The bottleneck is not that the node is “bugged” or that your CPU is weak — it’s that you’ve routed a genuinely heavy workload through the slower device, and the “text encoding is cheap” rule of thumb simply doesn’t apply at this model size.

This is exactly the trap that the umt5-xxl config falls into: the Wan reference implementation keeps this model parked on the CPU by design, which works in their data-center context where time per prompt is amortized over many gens — but on a single-user desktop machine, that per-prompt cost is something you feel every single time you hit Queue.

Why Force-to-GPU Backfires on a Single Card

Here’s where the well-meaning advice turns dangerous. The CLIPLoader (GGUF) node (from city96/ComfyUI-GGUF) exposes a device parameter, and ComfyUI also ships a built-in SelectCLIPDevice node that lets you route the CLIP encoder to CPU or a specific GPU. So you can absolutely force umt5 onto the GPU. The question is whether you should.

On a single-GPU machine the answer is usually no, for a simple reason: VRAM is shared. The same 16 GB (or 12 GB, or 8 GB) that now has to hold the 4–5 GB umt5 encoder is the same VRAM your Wan diffusion model needs to fit. Wan 2.2 at 14B, even heavily quantized with block-wise offloading, already runs at the edge of what a 16 GB card can hold. Carve another 4–5 GB out of that budget for a resident text encoder, and the UNET no longer fits — you’ll see the same OOM you were trying to avoid, plus slower swapping as ComfyUI’s offloader thrashes weights in and out.

The city96 README is blunt about this: the force-device feature is not intended for single-GPU users, and the maintainer warns specifically against setting it to cuda:0 and then complaining about OOM. It exists for genuinely multi-GPU layouts where you have a second (often older or slower) card you can dedicate to the encoder. Trying to “tune” a single GPU into hosting both a big encoder and a big diffusion model is a losing fight against physics.

How to Actually Decide: Match the Encoder to the Hardware

Rather than blindly forcing devices, diagnose against your real situation. There are three coherent outcomes, and which one is right depends on your VRAM headroom and whether you have a spare GPU.

1. Single GPU with tight VRAM (8–12 GB): leave it on CPU, shrink the encoder. The correct fix here is not to move the encoder to the GPU — it’s to make the CPU-side work cheaper. Drop from Q4 down to Q3_K_S or Q2_K quantizations (Wan encoder GGUFs are published across the quant range), and confirm you’re using the fp8 or GGUF build rather than the 11.4 GB bf16 original. Accept the one-time per-prompt cost; it’s a single pass, not 50.

2. Single GPU with headroom (16 GB+): aim for fp8, and let the default stand. If you have room, upgrade the encoder to the fp8 build (6.7 GB) and let ComfyUI decide placement based on the loader’s default. On newer ComfyUI versions with async weight offloading, the offloader will move the encoder in, use it once, and evict it before the UNET needs the space — so you get GPU-speed encoding without permanently surrendering VRAM.

3. Genuinely two GPUs: use SelectCLIPDevice or the GGUF loader’s device field. This is the only scenario where force-to-GPU is the right call — and it should target the second device (cuda:1), not the primary. Put the encoder on the spare card, keep cuda:0 clean for the diffusion model. Note the known multi-GPU quirk in ComfyUI-GGUF issue #96: device selection for CLIP and UNET can become entangled on real multi-GPU setups, and users have reported the UNET stubbornly reloading onto the wrong device until a full ComfyUI restart — so treat multi-GPU placement as its own tuning exercise, not a one-click swap.

There’s also a useful diagnostic sanity check before you change anything: watch what is slow. If the CLIPLoader or first KSampler step dominates and everything after it runs at normal speed, you’ve confirmed it’s the encoder’s one-time cost, and the decision above applies. If instead every step is equally slow, your encoder was never the issue — you have a different problem, like a CPU-offloaded UNET or a missing CUDA backend on the diffusion model itself. Correctly identifying the slow stage is half the battle (and it’s the same discipline we applied when tracing a 14B fp8 VRAM crash).

Verifying the Fix

After you make a change, confirm it with hard numbers rather than a feel. Check the ComfyUI terminal/console output for the load logs, which print which device each model actually landed on. Then time one generation from Queue to first progress-bar movement — that window is your per-prompt encoding cost. If you moved the encoder to a second GPU, that window should collapse; if you shrunk the quant instead, it should shrink modestly while your peak VRAM (visible in nvidia-smi or a VRAM-debug node) stays comfortably under your card’s limit. If peak VRAM suddenly spikes toward the full card after a force-to-GPU change, you’ve confirmed you stole memory the diffusion model needed — undo it and fall back to outcome 1 or 2.

The Takeaway

The next time someone tells you to “just force the text encoder to the GPU,” ask yourself which text encoder you’re actually running. For the small CLIP encoders in classic SD workflows, leaving them on the CPU is the correct, cost-free choice. For Wan’s umt5-xxl, the encoder is a real 11-billion-parameter model, and neither extreme — blind CPU or blind GPU — is automatically right. Diagnose first: identify the slow stage, know your encoder’s actual size, count your GPUs, and match placement to the VRAM you can afford to give up. That discipline beats force-fighting ComfyUI’s offloader every time.

You run a Wan2.2 image-to-video workflow in ComfyUI. The first render completes fine. The second one starts crawling. By the third, your system fans are screaming, free -h shows almost everything gone, and eventually the whole machine freezes or the process gets OOM-killed. You check nvidia-smi and VRAM looks reasonable — so why is RAM bloated and never coming back?

The instinct is to call it a memory leak. In most cases it isn’t. It’s ComfyUI holding on to node results on purpose, combined with a smart-memory manager that is slower to reclaim than you expect — and on Linux, an overcommit default that turns a slow bleed into a hard system freeze. This guide walks through diagnosing the real cause and fixing it with launch flags, node ordering, and OS-level guardrails.

Why ComfyUI holds RAM that looks “leaked”

ComfyUI caches the output of every node it executes so that, if you change a downstream node and re-run, it doesn’t have to recompute the expensive upstream work. That cache lives in system RAM (and sometimes VRAM) by default. On a short image workflow the cache is small. On a video workflow — especially Wan2.2 at higher frame counts — a single latent batch can be gigabytes, and a chain of cached intermediates quickly adds up to tens of GB.

The ComfyUI maintainers characterize this explicitly as efficient caching, not a leak. In GitHub issue #11301 (“RAM Memory Leak”), a maintainer replied that the growth “isn’t actually a memory leak, it’s just efficient caching” and pointed to the --cache-ram flag as the fix. That’s the key reframe: your RAM isn’t being wasted, it’s being retained, and you can cap it.

A second, related issue #9484 (“RAM not being fully released after WAN 2.2 generation”) — still open — reports Wan2.2 workflows that “previously ran without issues” starting to break after ComfyUI and node updates. Several reporters trace it to the cache and to a specific Torch version regression where cache-none throws errors until they downgraded to torch 2.7.1. So the problem has two faces: the cache retaining memory, and occasional upstream regressions in torch or specific nodes.

Step 1: Confirm it’s the cache, not a leak

Before changing anything, prove where the memory is going. On Linux, run this right after a video render and watch it across three consecutive runs:

free -h                # system RAM
echo "---"
nvidia-smi --query-gpu=memory.used,memory.total --format=csv  # VRAM

If RAM climbs steeply on run one, then stays flat at that elevated level on runs two and three, you’re looking at the cache reaching a steady state — not an unbounded leak. A true leak keeps climbing every run toward exhaustion. The distinction matters because it changes the fix entirely: a cache is tunable, a leak is a bug report.

You can confirm the cache hypothesis directly. Restart ComfyUI fresh (RAM drops back to baseline), load the exact same workflow, and watch memory climb to the same plateau without ever rendering anything new — cached results from earlier are being replayed. If you enabled --cache-classic or left the default in place, this plateau will be large on video workflows.

Step 2: Pick the cache mode that matches your RAM

ComfyUI’s cache flags are mutually exclusive — you pick one of --cache-ram, --cache-classic, --cache-lru, or --cache-none. From the official startup flags reference:

• --cache-none — reduced RAM/VRAM usage; re-executes every node on each run. Slowest, but the most deterministic memory floor.
• --cache-lru N — cache only the most recent N node results, evicting the oldest. A good middle ground on 16–32 GB machines.
• --cache-ram G — cap the cache at G gigabytes and let cached results spill to disk rather than memory. The direct answer to “it holds too much RAM.”
• --cache-classic — old, aggressive caching. Rarely what you want on a RAM-constrained box for video.

For a Wan2.2 video workflow on a 16 GB or 32 GB machine, --cache-lru 10 or --cache-ram 8 is the sweet spot: your RAM stops growing past a known ceiling, and you keep most of the speed benefit of caching. --cache-none is the nuclear option when you need the absolute most predictable floor and don’t mind slower re-runs.

Step 3: Order matters when you clear memory mid-workflow

If your workflow uses a “Clear Cache” or “Clean VRAM” node, its position in the graph changes what actually gets freed. A common community fix reported on r/comfyui is that clearing after the VAE decode — not before — is what reclaims RAM. The reasoning: the decode produces the large pixel tensors that are the biggest single cache entries. Clear the cache after you’ve handed those tensors off to the VAE encode/decode, not before generating them.

The practical ordering that works on video loops:

Generate → VAE Decode → Clear Cache / Clean VRAM → (next iteration)

Put the cleanup node at the tail of each loop iteration, after decode, so the largest intermediates are evicted before the next iteration allocates. A cleanup node placed before decode often frees nothing meaningful because the real memory spike hasn’t happened yet.

Step 4: The Linux overcommit trap (the actual freeze)

The most dangerous symptom of this whole problem isn’t the slow RAM creep — it’s the hard system freeze on Linux. Issue #9484 documents it directly: “the issue is worse in linux due to overcommitting RAM, causing a hard freeze when comfy fills the RAM completely.”

Linux defaults to vm.overcommit_memory=0, which hands out more memory than physically exists and hopes applications don’t all use it at once. When ComfyUI’s cache balloons and a second model load needs another few GB, the kernel overcommits, then has to reclaim under pressure — and on a desktop box with a GPU workload in flight, the reclaim can stall the GUI, the display manager, and even SSH long enough that the machine looks bricked.

Two remedies, in order of preference:

1. Cap it at the application layer (preferred): set a cache ceiling with --cache-ram or --cache-lru so ComfyUI simply never asks for unbounded RAM in the first place.
2. Constrain it at the OS layer (belt-and-suspenders): run ComfyUI inside a systemd scope or cgroup with a MemoryMax hard limit, so the kernel OOM-kills ComfyUI instead of the whole desktop. A systemd-oomd marker on the service adds a second, softer layer that reaps the offender before the system decides to.

The cgroup approach is what I use on a shared 27 GB RAM / 16 GB VRAM machine that has to keep a GUI and other services alive. ComfyUI launches through a wrapper that sets MemoryMax=14G and marks the unit for systemd-oomd; if a runaway video workflow tries to grab more, the cgroup is killed cleanly and the desktop stays responsive. The same idea works on any distro with systemd — you don’t need the whole server, just a scoped limit.

When it is a leak (and a quick win)

Not every case is the cache. The two signals that push you toward a real bug:

1. RAM climbs every single run with --cache-none set — there’s no cache left, so nothing should grow.
2. The symptom tracks a recent update. #9484 reporters pinned a regression to torch on the cu128 wheel and fixed it by pinning: pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 --index-url https://download.pytorch.org/whl/cu128. If your memory behavior changed after a ComfyUI, torch, or node update, downgrading the specific component is a fast, reversible test.

For the general “which node is responsible” question, the official ComfyUI guidance is to rule out custom nodes first: launch with --disable-all-custom-nodes (or the newer --disable-all-custom-nodes --whitelist-custom-nodes ComfyUI-Manager) and confirm whether the growth persists on a known-good workflow. If it disappears, the culprit is a custom node; re-enable them in batches to isolate it. The full procedure is documented in the custom-node troubleshooting guide.

How to verify you fixed it

After changing flags and node ordering, prove the fix with a three-run cold test:

# snapshot before
echo "== before =="; free -h | grep Mem; nvidia-smi --query-gpu=memory.used --format=csv,noheader
# render 1..3, snapshot after each
for i in 1 2 3; do
  echo "== after run $i =="
  free -h | grep Mem
  nvidia-smi --query-gpu=memory.used --format=csv,noheader
done

A successful result: RAM rises on run one, then plateaus on runs two and three instead of climbing without bound, and it sits comfortably below your physical limit. If you added a cgroup, deliberately over-request once (a huge frame count or batch) and confirm the cgroup is reaped while the desktop and SSH stay alive.

For more on squeezing ComfyUI into a low-RAM box, see our write-up on ComfyUI VRAM launch flags, and for the model-side of the same OOM fight, our guide to fixing CUDA out-of-memory in llama.cpp.

Conclusion

ComfyUI “not releasing RAM” after video generation is almost always the result cache doing its job too enthusiastically, not a leak. Diagnose by watching RAM plateau across runs, then cap it with --cache-ram or --cache-lru, clear memory after VAE decode, and — on Linux — protect the rest of the box with a cgroup MemoryMax so overcommit never freezes your whole machine. If the growth is unbounded under --cache-none or started right after an update, that’s your signal to bisect custom nodes and pin torch back. Fix the cache first, and most “memory leak” reports dissolve on their own.

One of the more confusing ComfyUI failures to hit a brand-new GPU is a Python traceback that ends in cuDNN Frontend error: [cudnn_frontend] Error: No valid execution plans built. It tends to appear the moment a workflow reaches the sampler, produces no image, and sends people down a rabbit hole of reinstalling PyTorch, cuDNN, and drivers — none of which actually fixes it. This guide explains what the error actually means, why it specifically targets RTX 50-series (Blackwell) cards, and the one launch flag that restores generation.

What the Error Actually Means

The error is not about a missing or broken cuDNN installation. It is about a mismatch between PyTorch’s default attention path and a GPU architecture that is newer than what that path knows how to handle. By default, PyTorch uses its scaled dot-product attention (SDPA) implementation, which can dispatch the computation to the cuDNN backend under the hood. On Blackwell-generation GPUs, that cuDNN backend has no registered execution plan for the new compute capability, so it returns the message you see in the traceback.

The binding nuance is worth spelling out because it explains a lot of failed “fixes.” Calling torch.backends.cuda.enable_cudnn_sdp(False) from a sitecustomize.py file does not work in this situation. The flag is not retained through CUDA initialization, so by the time PyTorch would honor it, the decision point has already passed. As the ComfyUI troubleshooting documentation states plainly, the only reliable fix is the launch flag.

Why It Hits Flux Hardest

With FLUX models specifically, the failure lands on the very first attention block of the UNet. That means the error is not a slow, intermittent degradation — it is an immediate hard stop. No image is ever produced, which makes the failure especially visible and often makes people think their model is corrupted, when the model is completely fine. The same underlying mechanism can affect any model that routes attention through the cuDNN SDPA backend on a Blackwell card, but FLUX makes it impossible to miss.

If your workflow produces nothing and the console shows only the No valid execution plans built line followed by a traceback that mentions execution.py around the sampler, this is very likely the cause rather than a broken checkpoint or invalid prompt.

!!! Exception during processing !!! cuDNN Frontend error: [cudnn_frontend] Error: No valid execution plans built.
Traceback (most recent call last):
  File "/home/user/ComfyUI/execution.py", line 496, in execute
  ...
RuntimeError: cuDNN Frontend error: [cudnn_frontend] Error: No valid execution plans built.

The Fix: –use-split-cross-attention

The recommended solution is to launch ComfyUI with a hand-written attention implementation that bypasses SDPA and its cuDNN backend entirely:

python main.py --use-split-cross-attention

This flag forces ComfyUI onto the split-cross-attention path, which uses an explicit attention implementation instead of letting PyTorch dispatch to the broken cuDNN backend. For desktop and portable installs, add the flag to whatever launcher you normally use:

• Manual install: run python main.py --use-split-cross-attention from the ComfyUI directory.
• Portable install: edit run_nvidia_gpu.bat (or create a copy) and add --use-split-cross-attention to the python invocation line.
• Desktop app: add the flag to the custom launch arguments in settings, or start the server manually with the flag.

Because the flag simply sidesteps the problematic backend rather than changing model or sampler settings, your FLUX (or any other) workflow should start producing images again without altering your quality settings, LoRAs, or sampling configuration.

The CUDA 12.8 Requirement Most People Miss

There is a second, related failure that often gets tangled up with this one. Blackwell cards will not run correctly with PyTorch builds compiled against older CUDA versions. As the NVIDIA developer forums and PyTorch discussions confirm, builds compiled against cu121 will not run on RTX 50-series cards at all. You need a PyTorch build compiled with CUDA 12.8 or newer to properly support the Blackwell compute capability.

This matters because the symptoms can look similar to the cuDNN error even though the root cause is different. If you are on a Blackwell card and see errors that reference an unsupported compute capability (for example sm_120 or sm_103) or experience a silent CPU fallback, check your PyTorch build’s CUDA version before anything else:

python -c "import torch; print(torch.__version__); print(torch.version.cuda)"

If the printed CUDA version is 12.1, you need to reinstall a CUDA 12.8+ build of PyTorch. Only after you are on a CUDA 12.8+ build should you expect the --use-split-cross-attention flag to be the remaining piece. These are two distinct problems with the same fix checklist, and treating them as one will waste time.

Diagnostic Checklist

Work through these in order. Each step either rules out a cause or pinpoints the exact one:

1. Confirm the card is Blackwell. An RTX 5070, 5080, or 5090 (or a laptop 50-series chip) is Blackwell. Older generations rarely trigger this specific cuDNN message.
2. Check your PyTorch CUDA build. Print torch.version.cuda. If it reads 12.1, upgrade to a CUDA 12.8+ build first; the cuDNN flag alone will not help you on a cu121 build.
3. Apply the launch flag. Start ComfyUI with --use-split-cross-attention and rerun the workflow.
4. Isolate from custom nodes. If it still fails, rule out custom nodes before blaming core ComfyUI.
5. Weigh SDPA quality vs compatibility. The split-cross-attention flag changes the attention path, so if you notice any subtle output difference, that is expected — but on Blackwell it is the path that actually works.

Why This Matters for Local AI Users

Blackwell cards are exactly the GPUs that local AI enthusiasts buy specifically to run model-heavy workflows like FLUX and video generation. The irony is that the newest hardware is precisely where the default attention path breaks, because the cuDNN backend has not yet caught up to the architecture. Understanding the difference between a driver problem, a PyTorch build problem, and a backend-dispatch problem is what separates a five-minute fix from an afternoon of reinstalling software.

The relevant facts here come from ComfyUI’s official troubleshooting documentation and from NVIDIA’s own developer forums, both of which identify the SDPA-to-cuDNN dispatch and the CUDA 12.8 build requirement as the two controlling variables. If you are running a Blackwell card and your FLUX workflow dies at the sampler, the flag in the next section is the fix that makes it work again.

Conclusion

The No valid execution plans built error is a Blackwell-specific dispatch problem, not a broken install and not a corrupted model. The fix is a single launch flag, --use-split-cross-attention, which routes attention through an explicit implementation instead of the cuDNN backend that has no plan for the new architecture. Pair it with a CUDA 12.8+ PyTorch build and the failure disappears. For the full official guidance, see the ComfyUI troubleshooting documentation, and if you are chasing related VRAM or offloading behavior, our guide to ComfyUI attention backends explains how the different paths trade speed, VRAM, and compatibility.

You paid for a 16 GB card, and you expect Wan 2.2 to use it. So it’s frustrating when you open a monitoring tool mid-generation and watch your VRAM sit at roughly 7–9 GB while generation crawls. You aren’t out of memory, you aren’t crashing — you’re just slow, and a chunk of your GPU is sitting idle. This is a real, reported issue, not a misconfiguration on your end, and it has a name and a fix path.

This article covers the recent GGUF offloading regression in ComfyUI, how to confirm you’re affected, the knobs that actually matter (blocks_to_swap, offload_device, and the launch flags), and a few diagnostics that will tell you which lever to pull. Everything here is grounded in upstream issue trackers and official documentation. Where I describe a setting or a symptom, I link to where it’s documented so you can verify it yourself before changing anything.

First, what changed: the deprecation of the WAN Block Swap node

For a long time, the WanVideoBlockSwap node in Kijai’s ComfyUI-WanVideoWrapper was the primary way to control how much of a Wan model stayed in VRAM versus being swapped to CPU memory. The node produced a block_swap_args dictionary that you handed to the model loader, letting you specify exactly how many transformer blocks to offload. This is documented in the node’s guide: the node lets you “specify a number of transformer blocks to swap to CPU memory, with additional settings to configure offloading of specific model components.”

That node has since been removed and deprecated in the current ComfyUI core path. The consequence is spelled out in ComfyUI issue #11081: after the Block Swap node was removed, GGUF models “now only use ~45–55% of VRAM (~7–9 GB on a 16 GB 4080) even in NORMAL_VRAM mode.” In other words, the manual offloading behavior that used to hit 80–90% VRAM is gone, and default automatic offloading is more conservative — leaving performance on the table.

This matters because of a subtle truth in the issue thread: GGUF in ComfyUI does not currently take advantage of async offload or memory pinning. One maintainer-adjacent commenter states it plainly: async offload “is not implemented for ComfyUI-GGUF,” and “neither is the recent memory pinning feature.” So a GGUF-loaded Wan 2.2 model falls back to a slower, more conservative memory path than a native fp8/fp16 load, and the deprecated Block Swap node used to be the workaround for exactly this.

How to confirm you’re affected

Before touching any setting, collect two numbers during an actual generation. Start ComfyUI with the frontend, load a Wan 2.2 GGUF workflow, and run a short video (for example, 640×480 at 81 frames, which is the configuration referenced in the issue thread). While it’s generating, watch VRAM in nvidia-smi or a panel like Crystools.

watch -n 1 nvidia-smi

Two signals, combined, point at the regression:

• VRAM stays in the 45–55% band on a card that previous versions filled to 80%+.
• Generation takes noticeably longer from a cold start than it used to, with a “calculating” or load phase before the model actually lands in VRAM.

The issue report also shows a useful diagnostic line from the console: a log entry like loaded partially: 3114.55 MB usable, 1106.95 MB loaded, 3404.00 MB offloaded. If you see a large “offloaded” figure alongside low utilization, that’s the conservative offloarding at work — the model is being pulled to CPU that a fuller card could hold.

The levers that actually move memory

blocks_to_swap and offload_device in WanVideoWrapper

If you want the manual control back that the deprecated node used to provide, the same concepts now live on the model loader itself. ComfyUI-WanVideoWrapper exposes a blocks_to_swap setting — the number of transformer blocks moved to CPU — alongside an offload_device option that controls where offloaded weights live (typically CPU vs. “main device”). The block-swap node’s own documentation describes the mechanism: “specify a number of transformer blocks to swap to CPU memory” to reduce VRAM usage.

The practical rule: if your card has VRAM to spare and you’re seeing the conservative half-utilization, reduce the number of blocks being swapped (or set blocks_to_swap to 0) so more of the model stays resident. The trade-off is always the same — fewer blocks swapped equals lower VRAM pressure removed but potentially higher peak VRAM. On a 16 GB card running the 14B model, keep an eye on nvidia-smi and back off one block at a time if you creep toward an out-of-memory error.

The –reserve-vram trick (documented, use with care)

The issue thread surfaces a genuinely useful hack: negative values for the --reserve-vram launch flag. ComfyUI reserves a chunk of VRAM and sizes the model load around that estimate; passing a negative value (for example --reserve-vram -1) “tricks” the current memory estimation into using the GPU more fully — with the author’s own caveat that it risks OOM if you overdo it. This is presented in the issue itself as “just a funny trick,” so treat it as a diagnostic, not a permanent config.

python main.py --reserve-vram -1

The flags to test when diagnosing

The issue reporter isolated the behavior by toggling two launch flags: --disable-pinned-memory and --disable-async-offload. Since GGUF already skips async offload and memory pinning, these flags are diagnostic rather than fixes, but they’re worth knowing. More importantly, the thread contains a maintainer note that pinned memory is “performance critical” — so if you’re on native (non-GGUF) weights, you generally want pinning on, not off.

Does this affect you, or is it something else?

It’s easy to misdiagnose slow generation. Before you conclude it’s the regression, rule out three non-causes:

• Thermal throttling. A card that’s hitting its thermal ceiling drops clocks, which looks identical to “slow” in the frontend. Check nvidia-smi for temperature and clock speed before blaming ComfyUI.
• The text-encoder tax. The issue thread notes the Wan text encoder “takes way longer with async offload turned on.” If your bottleneck is at the start of a run, it may be the encoder load, not the diffusion model.
• System RAM spillover. If your system RAM is small relative to the model, offloaded blocks and the model’s overflow land on a slow page cache. free -h during a run will tell you if you’re spilling to disk.

If none of those apply and you’re on a recent ComfyUI with a GGUF Wan 2.2 model that won’t fill your card, you’re most likely on the reported regression path.

A practical recovery sequence

Here’s a repeatable order of operations for a 16 GB card running Wan 2.2 GGUF:

1. Baseline. Run the workflow once and record VRAM utilization and wall-clock time. Watch nvidia-smi and note the peak and the average.
2. Check for the offload line. Look for the loaded partially / offloaded log line described above to confirm conservative offloading is active.
3. Reduce blocks_to_swap incrementally (toward 0) on the loader, re-running a short test after each change, until VRAM utilization rises toward a healthy band without tripping OOM.
4. Only if needed, test --reserve-vram -1 on a short run to see whether fuller utilization actually translates to faster wall-clock time — then revert unless the gain is meaningful and stable.
5. Verify stability with your real production resolution and frame count, since higher resolution and longer videos raise peak VRAM and can turn a “safe” setting into an OOM.

Conclusion

Slow Wan 2.2 GGUF generation on an under-filled GPU is not your imagination, and it’s not a driver problem you need to chase. It’s a documented offloading regression that followed the deprecation of the WAN Block Swap node, and it shows up as a card stuck at roughly half utilization. The fix is a matter of measuring first — VRAM, offload log lines, and wall-clock time — then pulling the right lever: blocks_to_swap, offload_device, and, when you understand the OOM risk, the --reserve-vram flag.

Because this is an active upstream issue, the behavior will likely change as the ComfyUI core improves GGUF offloading. Track issue #11081 and the official Wan2.2 memory optimization announcement for updates before you bake any workaround into your production launch script. If you’re already deep into video-generation tooling on this hardware, read our guide to running Wan2.2, CogVideoX, and Mochi on consumer hardware, and the piece on Wan2.2-Animate and what it means for local motion generation.

Note on evidence: this article is a documentation and source-based analysis of a reported upstream regression. No benchmark numbers here are invented — every figure (the ~45–55% utilization, the 640×480 / 81-frame reference, the offloaded log line, and the --reserve-vram trick) is quoted directly from the linked GitHub issue and official sources, so you can verify each claim for your own environment.

The Symptom: A 400 Error That Shouldn’t Happen

You are running llama-server locally, pointed at a solid quantized model, and everything works — until a conversation grows long. Then, mid-session, the OpenAI-compatible /v1/chat/completions endpoint returns an HTTP 400 Bad Request with a terse body:

the request exceeds the available context size, try increasing it

Your first instinct is probably to blame the model or the memory. Neither is the problem. What actually happened is that your prompt plus conversation history outgrew the context window you reserved at launch, and llama-server responded exactly the way it was rewritten to respond: it rejected the request instead of silently guessing which of your messages to throw away.

This is one of the most frequently reported llamacpp-server behaviors in the local-AI community, and it confuses people because it used to behave differently. Older builds would quietly truncate the oldest tokens for you. Newer builds do not. Understanding that shift — and what the --ctx-size and --context-shift flags actually do — is the difference between a flaky setup and a reliable one.

What --ctx-size Actually Reserves

The flag --ctx-size N (also written -c N, and exposed as n_ctx in the API) sets the size of the key/value (KV) cache — the running buffer that holds the tokens your model can “see” at once. Every history token and every generated token has to fit inside it. A larger context window means the model can remember more of a conversation, but it also means more system RAM and VRAM is committed up front, because llama.cpp reserves the KV cache as a fixed allocation rather than growing it per-request.

The important mental model: --ctx-size is a hard ceiling, not a soft suggestion. When the tokenizer expands your messages array and the total lands at or above that ceiling, the server does not have room to run the prompt. On modern builds it rejects the request outright. You can confirm the number a model actually supports by checking the reported context length when it loads (often called n_ctx_train in logs), and you should never set --ctx-size above a model’s native trained length unless you know the model supports RoPE scaling to extend it.

If you routinely hit the wall, the first and simplest lever is to raise the ceiling — provided you have the RAM to back it. Launching with a larger --ctx-size commits more memory, so before you bump it, verify how much headroom you actually have.

What --context-shift Does (and Doesn’t) Do

The most common misconception is that --context-shift is the flag that makes the server “auto-manage” a long conversation. It is not. Context shift operates during the generation phase: when the KV cache fills up while the model is still emitting tokens, context shift discards the oldest tokens and slides the window forward so generation can continue instead of stalling. That is a real and useful capability, but it is scoped to the completion step.

It does not rescue an incoming request whose prompt is already too large. A request that exceeds --ctx-size before a single token is generated fails validation before context shift ever gets a chance to run. This is exactly the scenario in the widely-tracked llama.cpp issue #17284, where users launching with both --ctx-size 131072 and --context-shift still received the 400 because the request itself — not the generation — overflowed the window.

Part of the confusion is historical. An older code path automatically truncated an overflowing prompt server-side. That logic was removed because it had side effects and made the server’s behavior unpredictable. The result is the current, consistent contract: the server tells you the request is too big and lets you (or your client) decide what to do.

Why the Server Hands Truncation Back to You

The maintainer’s position on this is explicit and worth quoting directly: “context truncation is the job of the client, not the server.” The reasoning is sound. The server does not know which of your messages are safe to discard. Dropping the system prompt would change the model’s behavior. Dropping the most recent user turn would break the request. Blindly deleting oldest-first can silently corrupt the very thing you asked it to preserve. Handling that correctly requires understanding your application’s semantics, which is information the server — correctly — does not have.

This is why tools like Ollama and LM Studio appear to “handle” long conversations automatically: the truncation and compaction is implemented in their client layer, not in the inference engine underneath. If you are driving llama-server directly through its API, you are the client, which means the responsibility lands on your code.

A Working Fix, Step by Step

The reliable fix has two parts: give the model a context window that matches what you actually send, and stop sending more history than the window can hold.

1. Size the window to the real workload

Start with a ceiling that comfortably fits your longest plausible exchange. If your typical conversation is under a few thousand tokens, a --ctx-size of 8192 or 16384 is usually enough and keeps memory low. If you genuinely need long-context work, raise it deliberately and verify the memory cost.

2. Compress history on the client

Before assembling the messages array, run your own token estimate and trim. A practical, robust pattern is to keep the system prompt always, keep the most recent user and assistant turns, and summarize or drop the middle. Many local stacks maintain a rolling summary buffer for exactly this reason. You do not need a perfect tokenizer; a conservative character-to-token estimate (len(text) / 4) gets you close enough to stay under budget.

3. Catch the 400 and recover

Because the error is well-defined, your client can treat it as a signal rather than a crash. On receiving the “exceeds the available context size” response, drop the oldest non-essential turns, re-send, and log the event. This turns a hard failure into a self-healing loop.

4. Confirm before you scale up

If you do choose to raise --ctx-size, check that you have the memory to back the reservation. A window that is too large will exhaust RAM and send the operating system into swap, which punishes every request with latency that looks like a separate, more mysterious performance bug.

Common Traps to Avoid

• Assuming --context-shift fixes prompt overflow. It only applies during generation. An over-limit request still returns 400.
• Setting --ctx-size beyond the model’s trained length. This inflates memory without adding usable context, and can produce degraded output unless the model supports length extrapolation.
• Blindly truncating oldest-first. Dropping the system prompt or recent turns silently changes behavior. Compaction must preserve the parts that matter.
• Ignoring the memory cost. A bigger context window is a fixed RAM/VRAM commitment, not a free upgrade.

How to Verify You Fixed It

Reproduce the failure deliberately: craft a request you know exceeds your current --ctx-size, confirm the 400, then apply your client-side truncation and confirm the same request succeeds. Watch the server log line for the request path, which will show 400 for the failure and 200 once your trimming keeps the prompt inside the window. A healthy setup is one where the server never has to reject a request because your client never sends one it cannot fit.

The takeaway is not to fight the 400. It is a correctly implemented guardrail telling you your client and your context budget are out of sync. Align them, and the error disappears — along with the mystery of why a “perfectly good” model stops answering long conversations. For more on how the context window and KV cache interact with GPU memory, see my earlier breakdown of the KV cache spilling into system RAM, and for the authoritative server reference consult the llama.cpp server documentation.

You set --ctx-size 131072 on a model that fits comfortably in your GPU, launch llama-server, and open nvidia-smi expecting the card to be busy. Instead you see the VRAM half-empty while your system RAM is being eaten alive — tens of gigabytes of it. Generation still works, but it is slow, and other desktop programs start swapping. The model is on the GPU, so what is eating all that RAM?

In almost every case the answer is the KV cache. This article explains why it happens, how to confirm it on your own machine, and the two flags — --cache-type-k and --cache-type-v — that pull the cache back onto the card. It is the same class of problem reported repeatedly in the llama.cpp community: a user runs a 27B model at 128K context with q8_0 cache and watches RAM fill despite free VRAM.

The Symptom: VRAM Free, RAM Exploding

The tell-tale signature is a mismatch between your two memory monitors. Run nvidia-smi and free -h side by side during a long generation. If you see the GPU reporting only a fraction of its VRAM used while the process keeps growing in system RAM as the context fills, you are looking at a KV cache that has spilled off the GPU.

This is easy to misread. People assume that if the weights fit in VRAM, everything should stay on the card. But llama.cpp allocates three distinct tenants in memory: the model weights, a working buffer and activations, and the KV cache. Each is sized independently, and the KV cache scales with your context length even when the model is tiny.

A 7B model at 128K context can easily need more memory for its KV cache than for its weights. When that cache no longer fits in the VRAM left over after the weights, llama.cpp quietly offloads the remainder to CPU RAM over the PCIe bus. There is no error and no warning — you just get slower tokens and a bloated free -h.

Why the KV Cache Grows so Fast

The KV cache stores the key and value tensors for every token in the context so that later tokens can attend to earlier ones without recomputing them. Its size is linear in context length and in the hidden dimension, so doubling your context doubles the cache.

The default cache data type is f16, meaning each key and each value element occupies 2 bytes. The rough budget for a single token’s KV (both K and V) is:

bytes_per_token = 2 bytes × 2 (K and V) × n_layers × n_kv_heads × head_dim

KV_cache_bytes = bytes_per_token × context_length

Because the per-token figure is large, a long context balloons fast. For a modern 27B-class model this can be several hundred bytes per token, so a 128K context translates to tens of gigabytes in the KV cache alone — more than the remaining VRAM on a 16 GiB or even 24 GiB card after the weights load. The fix is not to shrink context; it is to make each cache element cheaper via quantization.

The Fix: Quantize K and V Separately

llama.cpp lets you quantize the key cache and the value cache independently:

./llama-server \
  -m model.gguf \
  -ngl 99 \
  --ctx-size 32768 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0

The allowed values for both flags are f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, and q5_1. The two most useful in practice are q8_0 and q4_0.

q8_0 stores each element in 8 bits, roughly halving the cache size relative to f16. q4_0 stores each element in 4 bits, cutting it to a quarter of the f16 footprint. For a cache that was spilling tens of gigabytes into RAM, switching from f16 to q8_0 is frequently enough to pull the entire cache back onto the GPU. When you need the full 128K context and the card is genuinely tight, q4_0 on the value side (--cache-type-v q4_0) is the heavier lever.

There is a real robustness argument for quantizing K and V differently. The key cache tolerates lower precision better than the value cache for many tasks, so a common tuned configuration is --cache-type-k q8_0 --cache-type-v q4_0 — mild quantization on K, aggressive on V. It is the asymmetric-KV pattern discussed in the llama.cpp community, and it preserves more quality than blanketing both sides in q4_0 at the same memory cost.

Verifying the Cache Actually Stays on the GPU

Flipping the flags is only half the job. You need to confirm the cache is now resident in VRAM rather than assuming the flags worked:

• Compare before and after. Run the same generation at the same context with f16 and then with q8_0. Watch nvidia-smi and free -h. The RAM footprint of the process should drop sharply, and VRAM should rise.
• Look at startup logs. llama-server prints an estimate of the KV cache size and where it will live at launch. If the estimated size exceeds remaining VRAM, part of it is heading to RAM.
• Check tokens-per-second. A cache that lives in VRAM is much faster than one paged over PCIe. If your t/s jumps after the change, the cache has moved on-card.
• Watch for quality regressions. q4_0 is aggressive enough to degrade some outputs. If you depend on long-context precision, benchmark q8_0 against q4_0 on your own prompts rather than trusting rules of thumb.

One subtlety worth keeping in mind: --ctx-size sets the maximum, but the cache is allocated to match it. If you rarely use 128K of context but leave the ceiling at 128K, you are reserving KV memory you never touch. Where the workload allows, lowering --ctx-size to something realistic is a free win that often removes the spill entirely without touching precision.

Common Pitfalls

Three mistakes account for most of the confusion around this topic. First, people set -ngl 99 and assume that forces everything onto the GPU. -ngl controls layer offload only; the KV cache is sized independently and will spill regardless of how many layers are on the card.

Second, people conflate weight quantization with cache quantization. A model quantized to Q4_K_M still uses an f16 KV cache by default. Quantizing the GGUF does nothing for the cache, which is why you must set --cache-type-k and --cache-type-v explicitly even on a heavily quantized model.

Third, people chase the spill with more RAM instead of less cache. Adding system RAM does not fix the PCIe bottleneck; it just delays the swap storm. The performance-correct fix is to keep the cache on the card, which means quantization or a smaller context — not a bigger RAM budget.

Conclusion

A llama-server that fills system RAM while VRAM sits unused is almost never a bug in your driver or your install. It is the KV cache being sized for a long context with a high-precision default and quietly overflowing the card. The levers you actually control are the context ceiling and the cache data types.

Start with --cache-type-k q8_0 --cache-type-v q8_0, confirm the RAM footprint drops and the card fills, and only reach for q4_0 on the value side if you need the longest contexts on a genuinely small card. Verify each change against nvidia-smi, the launch estimate, and your own output quality — the flags are free to try, but the result depends entirely on your specific model, context, and card.

For the broader memory model behind layer offload and mmap flags — a different failure signature from this one — see our earlier walkthrough of llama.cpp CUDA out of memory and –n-gpu-layers. The authoritative flag reference is the llama-server(1) man page, and the llama.cpp repository tracks ongoing cache-quantization work such as asymmetric K/V and iq4_nl.

What Dynamic VRAM actually changed

If you updated ComfyUI recently and your once-stable workflow suddenly throws CUDA out of memory, hangs at Model Initializing..., or crashes a TensorRT node, the cause is almost certainly Dynamic VRAM. It shipped as the default memory system in stable ComfyUI and silently changed how every model weight is loaded, cached, and offloaded on your GPU.

Dynamic VRAM is ComfyUI’s replacement for the old “estimate-based” memory manager. Historically, ComfyUI tried to predict how much memory a model and its layers would need before running, then reserve enough free VRAM so the operation would not OOM mid-way. That prediction step was the source of a whole class of bugs: underestimated layers, aggressive offloading that thrashed between RAM and VRAM, and models getting pushed out to the Windows page file.

The new system drops prediction entirely. It is built on a custom PyTorch allocator called aimdo (the AI Model Dynamic Offloader) that loads model weights on demand, in the exact millisecond a tensor is needed for a calculation, and frees them the moment they are done. The Comfy team describes the core innovations in their announcement post and the deeper technical write-up at the comfy-aimdo repository.

Why the update suddenly causes OOMs and slowdowns

On paper Dynamic VRAM is strictly better: lower system RAM usage, no page-file paging, faster model loads, and the ability to run models larger than your physical RAM. The problem is that it also changed two behaviors a lot of people were relying on, and those regressions are what show up as “my workflow broke.”

1. Higher VRAM usage is now the norm

Dynamic VRAM deliberately keeps more of the model resident in VRAM because that is the fastest memory available. If you had tightened your workflow to run at 14.8–15.4 GB on a 16 GB card, the new allocator’s more aggressive caching can push you over the edge into a hard OOM. This is not a bug in your workflow; it is the allocator using the card more aggressively than the old manager did.

2. Cached weights stopped respecting the old flags

Several users on the official Dynamic VRAM discussion thread report that --reserve-vram no longer behaves the way it used to, and that model-unloading behavior changed even when they try to work around it. On an Intel Arc iGPU under Linux, one user reports workflows that used to run fine now OOM and get killed by the OS’s own out-of-memory killer rather than by ComfyUI’s graceful error path.

3. Third-party nodes that allocate their own VRAM break

The clearest, most reproducible breakage is with TensorRT-based nodes. ComfyUI_TensorRT and ComfyUI-Rife-Tensorrt allocate GPU memory through their own nvinfer1::DefaultAllocator, which fights with the new dynamic allocator. Users see an error like:

[E] [defaultAllocator.cpp::nvinfer1::internal::DefaultAllocator::allocate::53]
Error Code 1: Cuda Runtime

The same discussion thread also documents Expected all tensors to be on the same device errors on an RTX 5090 with 128 GB of system RAM, and measurable slowdowns of 2× or more on 4090/5090 servers using the Impact Pack and efficiency nodes — where the allocator keeps re-loading parts of the model that the old system would have left cached.

How to confirm Dynamic VRAM is your problem

Before reaching for the disable flag, run two quick checks so you are fixing the right thing.

1. Check your ComfyUI version. Dynamic VRAM is enabled by default in the latest git builds and in stable releases from roughly three weeks after the announcement. If you updated in that window and only then started seeing OOMs or slowdowns, the update is the most probable cause.
2. Compare against the old behavior. If the same workflow that ran cleanly yesterday now OOMs or takes dramatically longer, and you have not changed the workflow or the model, the memory manager is the variable that changed under you.
3. Watch for the telltale signature. “Model Initializing…” hanging, VRAM that fills more aggressively than before, and errors specifically from TensorRT-node allocators are the three most common fingerprints reported.

Note that --lowvram and --normalvram from the older memory model are separate concepts — if you are already managing VRAM with those launch flags, Dynamic VRAM sits underneath all of them and changes the baseline behavior those flags were tuned against.

The fix: disable Dynamic VRAM (and when not to)

The immediate, well-documented workaround is to launch ComfyUI with the disable flag, which reverts to the older estimate-based model loading:

python main.py --disable-dynamic-vram

How you add it depends on how you launch ComfyUI:

• Portable (Windows): edit run_nvidia_gpu.bat (or your launch batch file) and append the flag to the python main.py line, then save and relaunch.
• Manual install: add it directly to your python main.py launch command.
• Desktop app: the flag is passed through the app’s startup arguments. Several users report the Desktop app does not yet expose a clean toggle, so if you are on Desktop and need the flag, launching the server directly is the reliable path until a UI option ships.

There is also an inverse flag, --enable-dynamic-vram, which turns the feature on for systems where it is not auto-enabled — the official startup flags reference lists it as “auto on Nvidia, ROCm 7.14+”. The full flag documentation lives at docs.comfy.org.

The important nuance: do not disable it reflexively. On memory-constrained hardware, on Windows 11 with an NVMe SSD, and on single-GPU Nvidia setups, Dynamic VRAM is genuinely faster and lets you run larger models without paging. The people reporting problems are mostly on multi-GPU setups, TensorRT workflows, iGPU/Linux configurations, or servers where the old caching behavior was load-bearing for their specific node stack. If you are on a single RTX card doing straight image generation, the new default is very likely an improvement — measure your total workflow time (not it/s) before deciding.

What to do instead of just disabling it forever

The disable flag is a stopgap, and the Comfy team has stated they are actively fixing reported regressions (AMD support is next on the roadmap, and WSL is not planned at all). If you want the new allocator’s benefits without the breakage, here is a more durable path:

1. Isolate the culprit. Disable custom nodes one at a time — especially TensorRT nodes, the Impact Pack, and efficiency nodes — and re-run. On one documented case, disabling the TensorRT wrapper alone resolved the allocator conflict while keeping Dynamic VRAM on.
2. Update everything. The Comfy team recommends the latest PyTorch 2.10 with CUDA 13 for best performance under the new allocator. A stale torch build interacts poorly with it.
3. Test a clean install. One user on the discussion thread rolled back after calling the feature “buggy as hell and untested,” only to have problems recur — a clean reinstall with the proper requirements is worth trying before declaring it broken.
4. Report it properly. If the issue persists, the team explicitly asks for a detailed issue with full logs, the workflow, your hardware, and your OS. Performance reports should compare total workflow execution time, not it/s, because the allocator changes where time is spent.

Bottom line

Dynamic VRAM is a real, well-engineered improvement that shipped as a default and broke a minority of non-standard setups. The failure pattern is consistent: aggressive VRAM caching, a new allocator that conflicts with third-party allocators, and changed unloading behavior that the old flags no longer control. If your workflow regressed immediately after an update, --disable-dynamic-vram is the correct first step to confirm the cause, followed by isolating the specific node or upgrading your PyTorch rather than leaving the feature permanently off. Treat the flag as a diagnostic, not a permanent setting.

Note: this article is a documentation-based analysis of the shipped behavior, the official announcement, and the developer discussion thread. It is not a local benchmark — no performance numbers here are from tests run on this site’s hardware.

If your ComfyUI workflows suddenly started failing after a routine git pull or a desktop-app update, you are probably running into Dynamic VRAM — the new memory system that ComfyUI shipped stable in early spring 2026 and enabled by default. The feature is genuinely impressive on paper: it promises to slash system RAM usage, let you run models larger than your physical RAM, and eliminate out-of-memory crashes. But a sizable number of users are reporting the opposite: workflows that worked perfectly for months now throw OOM errors, TensorRT nodes crash with CUDA runtime failures, and Linux users are watching the OS kill their ComfyUI process.

This article explains what Dynamic VRAM actually changed, why it breaks specific setups, how to diagnose whether it is the cause of your problems, and the exact flag to turn it off. No speculation or faked benchmarks — every claim below is grounded in the official ComfyUI announcement, the official troubleshooting guide, and the official GitHub discussion where the maintainers are tracking regressions.

What Dynamic VRAM Actually Changed

To understand the failures, you need to understand what changed under the hood. The old ComfyUI memory model was predictive: before each inference step, it estimated how much memory every model in the graph would need, then did its best to keep that much free. When the estimate was too low, you got a classic “comfy tried to allocate” OOM. When it was too high, ComfyUI offloaded weights to CPU RAM despite your GPU having room to spare.

Dynamic VRAM replaces that whole approach with a custom PyTorch allocator the Comfy team calls the AI Model Dynamic Offloader, or aimdo. Instead of predicting, it reacts: model weights are placed into a virtual buffer (a “VBAR”) that costs zero physical VRAM, and the allocator offloads weights on demand only when the primary PyTorch allocator comes under real pressure. The official announcement cites several intended wins:

• Lower system RAM usage — weights are mapped as uncommitted, file-backed memory rather than deep-copied into PyTorch.
• No more page-file thrashing — cached weights are never pushed to the OS page file, so a 20 GB model no longer silently destroys your disk.
• Higher VRAM utilization — you will often see more VRAM used than before, which the team says is normal and intended.
• No model unload — ComfyUI now holds weights for the lifetime of the loaded model instead of unloading them back to RAM between runs.

That last change is the key to understanding most of the breakage. Under the old system, model weights moved from VRAM to RAM and back constantly. Under Dynamic VRAM, they stay put. For a single-model workflow that is faster and cleaner. For a multi-model workflow — especially one that mixes ComfyUI core with custom nodes that assume the old behavior — it can create exactly the OOM conditions the feature was meant to prevent.

Symptoms That Point to Dynamic VRAM

The first question is always: is this actually the cause? The following symptom set strongly indicates Dynamic VRAM is the culprit, especially when they appear together and immediately after an update:

1. OOM errors on a card that previously “just worked”

The most common complaint in the GitHub discussion is a 4090-class or larger card that ran a workflow fine for weeks, then started throwing “out of memory” after the update. One user with a 4090 running the klein9b model reported persistent OOM where before it was flawless. If your VRAM abundance did not change and your workflow did not change, the memory manager is the variable that moved.

2. TensorRT custom nodes failing with CUDA runtime errors

Dynamic VRAM breaks the allocator assumptions that TensorRT nodes rely on. Multiple users report that ComfyUI_TensorRT and ComfyUI-Rife-Tensorrt now fail at allocation time with a message like:

[E] [defaultAllocator.cpp] Error Code 1: Cuda Runtime
(in nvinfer1::internal::DefaultAllocator::allocate)

This is not a problem with your TensorRT install. It is the custom PyTorch allocator stepping in where TensorRT expects a standard CUDA allocation path.

3. Linux OOM-killer or Windows “low virtual memory” crashes

On Linux, several users describe ComfyUI and its terminal being killed by the kernel — not ComfyUI erroring out gracefully, but the OS stepping in to free memory. On Windows, the equivalent is a blue screen following an Event Viewer entry of “Windows successfully diagnosed a low virtual memory condition.” This class of crash is the most serious because it takes the whole session down, not just one generation.

4. “Expected all tensors to be on the same device” errors

One user on an RTX 5090 with 128 GB of system RAM — hardware that should be immune to memory pressure — hit Expected all tensors to be on the same device, but got index is on cuda:0 on load. This is a device-placement consistency bug tied to the new loading behavior, and disabling custom nodes does not fix it.

5. aimdo allocation-accounting errors

If your log shows repeated lines like aimdo_cuda_free_async: could not account free right before a crash, that is the Dynamic VRAM offloader itself failing to track memory — a bug in the new allocator, not in your workflow.

How to Confirm Dynamic VRAM Is the Cause

Don’t guess. Run a controlled check before you change anything permanently:

1. Note your ComfyUI version. Dynamic VRAM shipped stable in March 2026 and is on by default in recent git builds and the desktop app. If you recently updated, that is your window.
2. Reproduce with a minimal workflow. Load the default “first generation” workflow from ComfyUI. If that works but your real workflow fails, the trigger is a specific node or model-placement pattern in your graph.
3. Disable custom nodes. The official guide is explicit that most reports trace back to custom nodes. Temporarily disable them and re-run. If the error persists with core-only nodes, the issue is more likely the memory system itself than any single node.
4. Read the actual failure, not the summary. Click Show report on the “Prompt execution failed” dialog or read the terminal tail. Look for the specific strings above: defaultAllocator, aimdo_cuda_free_async, same device, or a kernel OOM-killer message.

If one or more of those signatures is present and your setup matches a reported case (large card, TensorRT, multi-model, or Linux), Dynamic VRAM is very likely the variable. The fix is a single launch flag.

Fixing It: The --disable-dynamic-vram Flag

The immediate, officially documented workaround is to launch ComfyUI with Dynamic VRAM disabled. From the startup flags reference, the command is:

python main.py --disable-dynamic-vram

Combine it with your existing flags as needed. For example:

python main.py --listen 0.0.0.0 --port 8188 --disable-dynamic-vram

Windows Portable users add the flag to the launch .bat file (for example run_nvidia_gpu.bat) rather than running it from a terminal. Desktop-app users should check the app’s advanced settings; if a flag field is not exposed there, fall back to a manual install or the portable build so you can control the launch line.

Why disabling is the right first response

Disabling reverts you to the previous estimate-based loading behavior — the one your workflow already ran on successfully. That is a low-risk rollback, and it instantly tells you whether Dynamic VRAM was the cause: if your workflow works again with the flag set, you have confirmed the diagnosis and gained a functioning setup while the maintainers stabilize the feature.

Two important caveats from the discussion:

• --reserve-vram may no longer be respected. Some Linux users report that even with --disable-dynamic-vram, the new loading behavior keeps models resident and no longer unloads them, which can still lead to crashes on constrained systems. If disabling alone is not enough, combine it with a conservative --lowvram or --normalvram mode appropriate to your card.
• Multi-GPU and SageAttention setups are affected separately. Users on multi-GPU systems report breakage that disabling Dynamic VRAM only partly resolves. If you depend on multi-GPU or SageAttention, check the discussion thread for the latest state before assuming a single flag fixes you.

Common Pitfalls When Working Around This

Avoid these mistakes, which show up repeatedly in the threads:

1. Comparing it/s instead of total wall-clock time. Dynamic VRAM can change the per-step timing. The maintainers explicitly ask users to compare total workflow execution time, because a lower it/s does not necessarily mean a slower workflow when weight loading is included.
2. Assuming higher VRAM usage means a leak. The official guidance is that higher VRAM occupation is expected and healthy under the new system. Don’t chase it as a bug.
3. Reinstalling everything on a hunch. One user did a clean install and still hit OOM. The problem is the memory system, not a corrupted install. Diagnose first, reinstall last.
4. Relying on the page file as a safety net. One of Dynamic VRAM’s design goals is to stop using the page file. If you had a bloated page file masking memory errors before, those errors will now surface. Fix the root cause instead of re-enlarging the page file.
5. Forgetting that WSL and AMD are unsupported. The feature is for NVIDIA on native Windows and Linux only. WSL is explicitly not planned, and AMD support is listed as a future step. If you run either, your experience does not reflect the intended behavior.

Verifying Your Fix and Deciding What to Keep

After applying --disable-dynamic-vram (plus any mode flag), verify deliberately rather than assuming:

1. Re-run the exact workflow that failed, not a trimmed-down copy. Confirm it completes end to end multiple times — OOM and allocator errors are often intermittent.
2. Watch the terminal, not just the image. A clean completion with a readable image but error spam in the log is still a broken state.
3. If TensorRT is your blocker, confirm your TensorRT nodes load and allocate without the defaultAllocator error after disabling.
4. On Linux, keep an eye on dmesg for OOM-killer entries rather than trusting that a single successful run means the kernel will behave.

Your end state should be a deliberate choice, not a knee-jerk rollback you keep forever by accident:

• If Dynamic VRAM works for you and you are on a supported NVIDIA/native setup with no TensorRT dependency, leave it on and enjoy the lower RAM usage and faster loads.
• If it breaks your workflow, keep --disable-dynamic-vram pinned in your launch script or .bat file, and subscribe to the GitHub discussion so you know when the specific regression affecting your setup is resolved.
• If you are uncertain, pin the flag now and test the feature again after the maintainers ship a few rounds of regression fixes — the discussion thread makes clear this is an actively patched, fast-moving area.

Conclusion

Dynamic VRAM is a real engineering improvement that will genuinely help most memory-constrained NVIDIA users — it is not a broken feature. But enabling a custom allocator by default means every workflow that depended on the old memory behavior is now a potential regression, and the OOM, TensorRT, and multi-GPU failures reported in the official discussion are that playing out in practice.

The practical takeaway is simple: learn the one flag that takes you back to known-good behavior (--disable-dynamic-vram), use it to confirm your diagnosis, then decide with data whether to stay on or off. If you have not yet read the underlying launch flags, our earlier walkthrough of ComfyUI’s VRAM flags (--lowvram, --normalvram, and friends) is the natural companion piece for tuning the fallback here.

Why “OpenAI-Compatible” Is a Promise, Not a Guarantee

The phrase “OpenAI-compatible API” is one of the most overused labels in the local-AI ecosystem. Ollama advertises it, llama.cpp’s built-in llama-server advertises it, and vLLM advertises it. The implication is seductive: swap out one base_url in your existing automation, point it at a box in your basement, and every call that worked against OpenAI’s cloud just works locally.

In practice, compatibility is a spectrum. A POST /v1/chat/completions endpoint that accepts a messages array and returns a choices[0].message.content string can legitimately be called “OpenAI-compatible” while still failing downstream automation in three very specific, very common ways: streaming tool calls that don’t parse, function calling that silently ignores your schemas, and chat templates that reformat your conversation in ways the client didn’t expect.

This article walks through those three failure modes as they manifest specifically with llama-server, the HTTP server bundled with llama.cpp. I’ll document what the official server docs actually promise, where the label breaks down, and what concrete flags and checks close the gap. This is a documentation-level analysis grounded in the official llama.cpp server README and the linked issue trackers, not a set of fabricated benchmarks — I’ll be explicit about what I verified by reading source versus what I’m reasoning about.

Symptom 1: Streaming Works Until the Model Tries to Call a Tool

The most common and most confusing breakage happens at the intersection of two features that each work fine in isolation: streaming and tool calling. Your plain chat streaming works perfectly — content chunks arrive with stable delta objects. Then you hand the client a tools array, the model decides to call a function, and the stream terminates with an error or, worse, returns a tool call buried in a string field the SDK can’t parse.

The root cause is a format mismatch, not a bug in your client. The OpenAI Python SDK expects tool calls to arrive as structured tool_calls deltas with typed function.arguments. Many models served through llama.cpp emit their tool calls in an XML or text-based format embedded at the end of the content stream. When that raw XML lands in a delta.content field, the SDK has no idea it’s supposed to be a structured call, and the automation breaks.

The fix has two parts. First, confirm the server is even rendering tool-call tokens through the correct Jinja path — function calling requires the --jinja flag, and in many cases a matching --chat-template-file override so the model’s tool-call syntax matches what the server’s parser expects (this is stated directly in the server README). Second, pin your client’s expectations to whatever the server actually returns: inspect the raw delta objects once, before you trust the abstraction. If the model emits XML tool calls, either switch to a model/GGUF whose bundled template emits JSON, or parse the text field yourself rather than assuming the SDK’s typed path will work.

Symptom 2: Function Calling Is Silently Ignored

A subtler failure looks like nothing went wrong at all. You pass a tools array with a JSON schema, the model responds in fluent natural language describing what it would do, and no tool_calls object appears anywhere. There’s no error — just a model that refuses to behave like a function-calling model.

This usually traces back to one of three misconfigurations, and they’re all worth checking before you blame the model:

• The server wasn’t started with tool support. llama-server exposes tools through a --tools flag. Start with --tools all to enable every supported tool, or pass a comma-separated allow-list like --tools name1,name2 to restrict which tools a request may use. Without this, function calling is not active even if your request body is perfectly formed.
• The chat template doesn’t express tool calls. Function calling is a property of the model’s chat template. A model whose bundled template has no tool-call syntax can’t produce a parseable call regardless of what schema you send. The override path — --jinja plus a tool-aware --chat-template-file — is frequently the missing piece.
• The schema is under-constrained. If you rely on JSON-mode or function-call output without a grammar or JSON-schema-constrained sampling, small models in particular will drift into free text. The server supports JSON-schema-constrained sampling, which keeps the generated output inside your schema instead of letting it collapse into prose.

The diagnostic order that saves the most time: read the raw response once (does a tool_calls array exist at all?), then check the server startup flags, then check the model’s actual chat template on the Hugging Face model card. One of those three will almost always be the culprit.

Symptom 3: Chat Templates Reformat Your Conversation Behind Your Back

The third failure mode is the one that produces “works with model A, breaks with model B” bugs that are maddeningly hard to pin down. Every instruction-tuned model ships with a chat template — a Jinja snippet that defines how a list of messages is collapsed into the raw prompt the model actually sees. Different model families format system prompts, tool calls, and assistant turns differently.

The server applies this template automatically, which is normally what you want. But it means the messages array you send is not the token stream the model receives. When you swap models — say from a ChatML-family model to a Qwen or DeepSeek variant — the same client code may silently produce different results because the template changed shape. System prompts that were once honored may be truncated, tool tokens may be rendered in a dialect the parser no longer recognizes, and multi-turn history may be folded differently.

The fix is to stop treating the template as magic and start treating it as a controllable dependency. For automation that must be stable across models, you generally want to pin the template explicitly via --chat-template-file rather than relying on the model’s bundled default, and to log what template is actually active in each deployment. The server README and the associated tool-calling discussions make clear that the “right” template is often a deliberate override choice, not something you can assume is correct out of the box.

Routing Multiple Models Through One Endpoint

Once you’re past the three compatibility traps, llama-server has a genuinely useful capability that’s worth understanding because it changes how you architect a small self-hosted stack: multi-model routing through a single server. You don’t need a separate process or port per model. You include a model field in the request body and the router dispatches to whichever model you name:

{
  "model": "ggml-org/gemma-3-4b-it-GGUF:Q4_K_M",
  "messages": [{"role": "user", "content": "hello"}]
}

For GET endpoints like /props or /metrics, the router reads the model query parameter instead. This is a meaningful simplification over running three servers on three ports and maintaining your own load balancer — but it also concentrates your compatibility risk in one place, because every routed model needs to tolerate the same client’s assumptions about templates and tool-call formats.

As with everything in this article, the practical takeaway is the same: verify the contract with a throwaway call before you wire it into production automation. The abstraction is real, but the edge cases are model-specific.

Verification Checklist Before You Ship Automation

Here’s the concrete checklist I’d run against any self-hosted OpenAI-compatible endpoint before trusting it in a pipeline:

• Plain non-streaming call — confirm a well-formed choices[0].message with a clean content string.
• Streaming call — confirm delta chunks arrive and terminate cleanly with a [DONE]-style finish.
• Tool call, non-streaming — confirm a structured tool_calls array (not text). If it’s text, you have a format mismatch to fix with --jinja and a template override.
• Tool call, streaming — confirm tool-call deltas parse; this is the highest-failure case and the one that most commonly forces a client-side text-parse fallback.
• JSON-schema-constrained output — confirm the response actually conforms to your schema rather than drifting into prose.
• Flag audit — log the exact server args in each deployment so you can reproduce a routing or tool behavior later.
• Template pinning — record which --chat-template-file (if any) is active per model.

Running this checklist is cheap compared to debugging a production automation failure that only manifests when a model happens to call a tool mid-stream.

Conclusion

“OpenAI-compatible” means “speaks a dialect of the same HTTP contract,” not “behaves identically to OpenAI’s cloud.” The three places that contract reliably bends — streaming tool calls, silently-ignored function calling, and chat-template reformatting — are all solvable, but only if you stop trusting the label and start verifying the actual bytes. The flags exist (--jinja, --tools, --chat-template-file, JSON-schema-constrained sampling), and the routing capability is genuinely useful. The work is in confirming which of them your specific model needs.

If you’re running any local inference alongside ComfyUI-style workflows, the same discipline applies there too — the gaps between “documented” and “actually works on your box” are where automation usually dies. For more on that side of local AI operations, see our guide to building a Python automation harness for ComfyUI, and for the broader picture of running video models on consumer hardware, read our breakdown of the open-source video stack.