The Symptom: 100% CPU, GPU Sitting Idle
You download a quantized Wan 2.2 text encoder — something like umt5-xxl-encoder-Q3_K_S.gguf — wire it into your workflow through a CLIPLoader (GGUF) node, hit Queue, and watch your CPU peg at 100% while the taskbar shows your GPU twiddling its thumbs. Encoding a single prompt takes what feels like an eternity, and you’re left wondering whether you bought the wrong hardware.
The instinctive fix, shared all over Reddit and Discord, is to “just force the CLIP loader onto the GPU.” On paper it makes sense: the GPU is idle, the CPU is choking, so move the work across. But before you reach for a SelectCLIPDevice node or a MultiGPU wrapper, you need to understand why that node routes to the CPU in the first place — because on a single-GPU machine, forcing it to cuda:0 is often the quickest way to trade a slow-but-working render for an outright out-of-memory crash.
Why the Text Encoder Runs Once — and Why That Matters
Text encoding in ComfyUI is not a per-step operation. Unlike the diffusion UNET, which runs its forward pass once for every single sampling step (anywhere from 20 to 60+ times per generation), the text encoder runs exactly once per prompt, at the very start of the pipeline. It converts your tokens into conditioning embeddings, and those embeddings are then reused for every subsequent step.
This is the foundation of the standard advice that “text encoding is fine on the CPU, it barely costs you anything.” For the small CLIP encoders used by SD1.5 and SDXL, that advice is perfectly correct: a ~120 MB CLIP-L encoder can churn through a prompt in a fraction of a second on any modern CPU, and keeping it off the GPU freees precious VRAM for the model that actually dominates runtime — the UNET.
But that reasoning silently assumes a small encoder. With Wan 2.2 and the umt5-xxl family, it completely falls apart, and this is the root of the confusion you’re running into.
The Real Cause: umt5-xxl Is a Full LLM, Not a Tiny CLIP
The Wan family does not use a dinky CLIP text encoder. It uses umt5-xxl, a massive encoder-decoder language model whose encoder alone is roughly 11.4 GB at bfloat16. Even after aggressive quantization, it stays large: the official fp8 build (umt5-xxl-enc-fp8_e4m3fn.safetensors) is about 6.7 GB, and a GGUF Q3_K_S quant still lands in the 4–5 GB range. For reference, that single text encoder is larger than some entire SD1.5 checkpoints.
So when you load a quantized umt5 through CLIPLoader (GGUF) and it lands on the CPU by default, you’re asking a consumer CPU to perform a full forward pass through an 11-billion-parameter-class model. Of course it pegs at 100% and takes a while. The bottleneck is not that the node is “bugged” or that your CPU is weak — it’s that you’ve routed a genuinely heavy workload through the slower device, and the “text encoding is cheap” rule of thumb simply doesn’t apply at this model size.
This is exactly the trap that the umt5-xxl config falls into: the Wan reference implementation keeps this model parked on the CPU by design, which works in their data-center context where time per prompt is amortized over many gens — but on a single-user desktop machine, that per-prompt cost is something you feel every single time you hit Queue.
Why Force-to-GPU Backfires on a Single Card
Here’s where the well-meaning advice turns dangerous. The CLIPLoader (GGUF) node (from city96/ComfyUI-GGUF) exposes a device parameter, and ComfyUI also ships a built-in SelectCLIPDevice node that lets you route the CLIP encoder to CPU or a specific GPU. So you can absolutely force umt5 onto the GPU. The question is whether you should.
On a single-GPU machine the answer is usually no, for a simple reason: VRAM is shared. The same 16 GB (or 12 GB, or 8 GB) that now has to hold the 4–5 GB umt5 encoder is the same VRAM your Wan diffusion model needs to fit. Wan 2.2 at 14B, even heavily quantized with block-wise offloading, already runs at the edge of what a 16 GB card can hold. Carve another 4–5 GB out of that budget for a resident text encoder, and the UNET no longer fits — you’ll see the same OOM you were trying to avoid, plus slower swapping as ComfyUI’s offloader thrashes weights in and out.
The city96 README is blunt about this: the force-device feature is not intended for single-GPU users, and the maintainer warns specifically against setting it to cuda:0 and then complaining about OOM. It exists for genuinely multi-GPU layouts where you have a second (often older or slower) card you can dedicate to the encoder. Trying to “tune” a single GPU into hosting both a big encoder and a big diffusion model is a losing fight against physics.
How to Actually Decide: Match the Encoder to the Hardware
Rather than blindly forcing devices, diagnose against your real situation. There are three coherent outcomes, and which one is right depends on your VRAM headroom and whether you have a spare GPU.
1. Single GPU with tight VRAM (8–12 GB): leave it on CPU, shrink the encoder. The correct fix here is not to move the encoder to the GPU — it’s to make the CPU-side work cheaper. Drop from Q4 down to Q3_K_S or Q2_K quantizations (Wan encoder GGUFs are published across the quant range), and confirm you’re using the fp8 or GGUF build rather than the 11.4 GB bf16 original. Accept the one-time per-prompt cost; it’s a single pass, not 50.
2. Single GPU with headroom (16 GB+): aim for fp8, and let the default stand. If you have room, upgrade the encoder to the fp8 build (6.7 GB) and let ComfyUI decide placement based on the loader’s default. On newer ComfyUI versions with async weight offloading, the offloader will move the encoder in, use it once, and evict it before the UNET needs the space — so you get GPU-speed encoding without permanently surrendering VRAM.
3. Genuinely two GPUs: use SelectCLIPDevice or the GGUF loader’s device field. This is the only scenario where force-to-GPU is the right call — and it should target the second device (cuda:1), not the primary. Put the encoder on the spare card, keep cuda:0 clean for the diffusion model. Note the known multi-GPU quirk in ComfyUI-GGUF issue #96: device selection for CLIP and UNET can become entangled on real multi-GPU setups, and users have reported the UNET stubbornly reloading onto the wrong device until a full ComfyUI restart — so treat multi-GPU placement as its own tuning exercise, not a one-click swap.
There’s also a useful diagnostic sanity check before you change anything: watch what is slow. If the CLIPLoader or first KSampler step dominates and everything after it runs at normal speed, you’ve confirmed it’s the encoder’s one-time cost, and the decision above applies. If instead every step is equally slow, your encoder was never the issue — you have a different problem, like a CPU-offloaded UNET or a missing CUDA backend on the diffusion model itself. Correctly identifying the slow stage is half the battle (and it’s the same discipline we applied when tracing a 14B fp8 VRAM crash).
Verifying the Fix
After you make a change, confirm it with hard numbers rather than a feel. Check the ComfyUI terminal/console output for the load logs, which print which device each model actually landed on. Then time one generation from Queue to first progress-bar movement — that window is your per-prompt encoding cost. If you moved the encoder to a second GPU, that window should collapse; if you shrunk the quant instead, it should shrink modestly while your peak VRAM (visible in nvidia-smi or a VRAM-debug node) stays comfortably under your card’s limit. If peak VRAM suddenly spikes toward the full card after a force-to-GPU change, you’ve confirmed you stole memory the diffusion model needed — undo it and fall back to outcome 1 or 2.
The Takeaway
The next time someone tells you to “just force the text encoder to the GPU,” ask yourself which text encoder you’re actually running. For the small CLIP encoders in classic SD workflows, leaving them on the CPU is the correct, cost-free choice. For Wan’s umt5-xxl, the encoder is a real 11-billion-parameter model, and neither extreme — blind CPU or blind GPU — is automatically right. Diagnose first: identify the slow stage, know your encoder’s actual size, count your GPUs, and match placement to the VRAM you can afford to give up. That discipline beats force-fighting ComfyUI’s offloader every time.
Leave a Reply