How to Run a 235B MoE Model on a 16GB GPU: –n-cpu-moe and –override-tensor Explained

The Problem: A Model That Shouldn’t Fit

Qwen3-235B-A22B sounds like it needs a rack of H100s. Read the model card and you’ll see 235 billion total parameters — on paper that’s a ~130 GB GGUF file in Q4_K_M quantization, nowhere near a 16 GB consumer card. Yet people run it at usable speeds on a single RTX 4060 Ti or 3090.

The reason is in the fine print: it’s a mixture-of-experts (MoE) model, not a dense one. Of those 235B parameters, only 22 billion are activated per token. The model has 94 layers and 128 “expert” feed-forward networks per layer, but the router only consults 8 of them for any given token. That means ~90% of the weights sit idle on any forward pass.

llama.cpp exposes two flags — --n-cpu-moe and --override-tensor — that let you exploit this sparsity directly. Instead of the crude all-or-nothing layer offloading that --n-gpu-layers gives you, these flags let you strand the expert tensors in system RAM while keeping the small “always active” parts of the model on the GPU. This article is a documentation-based walkthrough of what those flags actually do, where the common guides get them wrong, and how to verify your own configuration. No benchmarks are fabricated here — everything below is drawn from the llama.cpp source, its CLI reference, and the official Hugging Face model cards, and is reproducible on your own hardware.

First, Understand the MoE Memory Layout

Before touching a flag, you need to understand why offloading experts is different from offloading layers. In a dense model like Llama-3.3-70B, every one of the 70 billion parameters participates in every token. Offload any of it to CPU RAM and you slow down every single token — there is no “cheap” weight to strand.

An MoE model splits each transformer layer into two kinds of weights:

• Always-active parameters — the attention module, the embedding, the router, and a small “shared expert” FFN that fires on every token. These are relatively tiny (a few percent of total parameters) but they’re touched on every token, so they want to live on the GPU.
• Routed expert FFNs — the bulk of the parameters. For Qwen3-235B-A22B, each layer has 128 experts but only 8 activate per token. These tensors are enormous individually but each one is used rarely. This is the part you can afford to keep on slower CPU RAM.

In the GGUF tensor naming, the routed experts show up per layer as blk.N.ffn_gate_exps.weight, blk.N.ffn_up_exps.weight, and blk.N.ffn_down_exps.weight. The shared expert is blk.N.ffn_shexp.weight. This naming is the key to understanding how the offload flags work.

Flag #1: --n-cpu-moe N

The official CLI reference describes --n-cpu-moe N (short -ncmoe) as: “keep the Mixture of Experts (MoE) weights of the first N layers in the CPU.” Its sibling --cpu-moe (-cmoe) keeps all MoE weights on CPU.

Two details here matter and are frequently misreported:

1. It counts layers, not experts. Some community guides describe --n-cpu-moe 20 as “offload 20 of the 28 experts,” which is backwards. The flag moves the expert FFN weights of the first N layers to CPU. The attention, router, and shared expert of those layers stay wherever --n-gpu-layers put them.
2. It is a blunt per-layer cutoff, like --n-gpu-layers but specialized to expert tensors. You can’t use it to keep, say, only the attention on CPU while experts of the same layer stay on GPU — the whole layer’s expert block moves together.

Here’s a minimal launch that loads everything possible to the GPU but strands the expert FFNs of the first 40 layers on CPU:

./llama-server -m Qwen3-235B-A22B-Q4_K_M.gguf \
  --n-gpu-layers 99 \
  --n-cpu-moe 40 \
  -c 8192

--n-gpu-layers 99 (or -ngl 999) requests “push everything possible to the GPU.” --n-cpu-moe 40 then overrides that for the expert tensors of the first 40 layers, sending them to system RAM instead. The result: attention, router, and shared experts stay on GPU where they’re cheap to compute, while the huge rarely-touched expert matrices live in RAM.

The trick barely helps a dense model — there’s no sparse expert tensor to strand — so check the model card first. If it doesn’t say “mixture-of-experts,” --n-cpu-moe won’t save you.

Flag #2: --override-tensor (surgical control)

If --n-cpu-moe is a blunt layer cutoff, --override-tensor (short -ot) is a scalpel. It takes a regex that matches individual tensor names and assigns them to a specific buffer type. The syntax is pattern=buffer, with multiple patterns joined by commas.

The original PR that added it (llama.cpp #11397) offers the canonical MoE example:

./llama-server -m Qwen3-235B-A22B-Q4_K_M.gguf \
  --n-gpu-layers 99 \
  -ot "exps=CPU"

The pattern exps matches every routed-expert tensor (ffn_gate_exps, ffn_up_exps, ffn_down_exps) and sends them to CPU, while the shared expert (ffn_shexp) and everything else stays on GPU. That’s the cheapest way to reclaim VRAM, because the routed experts are the bulk of the model and the most rarely activated.

For finer control you can target specific layers. To offload only layers 40–47’s expert FFNs (the “late layers” that activate least often) while keeping early-layer experts on GPU:

-ot "blk\.(4[0-7])\.ffn_(gate|up|down)_exps\.weight=CPU"

Two gotchas that trip people up:

• Multiple -ot flags are deprecated. Passing -ot twice can silently send all matched experts to the last target. Combine everything into a single comma-separated regex instead.
• The pattern is a regex, and it’s easy to typo into matching nothing. Launch once with -ot ".*=CPU" and the -v (verbose) flag to print every tensor name being overridden, then build your real pattern from the actual names.

The Misinformation You Should Ignore

Because this is a fast-moving area, a few widely-circulated guides now contradict the current llama.cpp documentation. Two claims in particular deserve correction:

1. “You must add --split-mode row for --n-cpu-moe to work.” The current docs/multi-gpu.md marks row as deprecated, superseded by tensor. And critically, --split-mode tensor is not implemented for MoE architectures — the docs list Grok, DeepSeek2, Nemotron-H, and other MoE/hybrid families as explicitly unsupported, and attempting it fails with LLAMA_SPLIT_MODE_TENSOR not implemented for architecture '...'. On a single GPU the whole question is moot: --split-mode governs multi-GPU splitting. For one 16 GB card, don’t pass it at all.

2. Conflating “experts” with “layers” in the --n-cpu-moe argument. The flag counts layers. Treating it as an expert count produces a configuration that’s off by a factor, and authors then publish numbers that don’t reproduce.

The safe rule: for a single-GPU setup, prefer --n-cpu-moe for a quick fit and -ot for a deliberate one. Leave --split-mode alone.

Diagnostic Steps: Get It Right the First Time

Offloading is a memory-budget problem, and the fix is to measure rather than guess. Work through this order:

1. Record the baseline. Run nvidia-smi before launching and note free VRAM. Subtract a few hundred MiB for driver overhead.
2. Start conservative. Load with --n-gpu-layers 99 --cpu-moe (all experts on CPU) and a modest -c 4096. Confirm the server boots and produces tokens. This is the guaranteed-fit floor.
3. Reclaim experts in layers. Replace --cpu-moe with a smaller --n-cpu-moe N, or move from a blanket exps=CPU to a per-layer -ot pattern that only offloads the late layers. Watch nvidia-smi after each change and stop when used VRAM settles near ~80% of capacity, leaving headroom for the KV cache and activations.
4. Add context last. Only after the weight split is stable should you widen -c. The KV cache is the classic hidden variable — it grows with context length and can blow the budget even when the weights fit.

Remember the quantization math too: a Q8_0 or F16 file is two to four times larger than its Q4_K_M sibling. The same offload settings that work for Q4_K_M will OOM on Q8_0, so know your file’s footprint before choosing values.

Common Pitfalls and How to Verify

Several failure signatures have specific causes:

• OOM at startup or during prefill. Too many expert layers still on GPU, or too large a context. Lower -c first, or increase --n-cpu-moe.
• Decode barely faster than full CPU. Your -ot regex matched too much — likely attention or shared-expert tensors rather than only exps. Re-verify with -v that it hits only the routed experts.
• Runs, but VRAM slowly climbs toward the ceiling. The KV cache is growing across a long conversation. Size it explicitly or cap context.

To confirm a healthy configuration, look for three things:

1. Stable VRAM. nvidia-smi should show a plateau during generation, not a steady climb.
2. No swap churn. Watch free -h and vmstat; constant swap means the CPU-resident expert count is too high for your RAM.
3. Consistent throughput. Run the same short prompt several times — wide variance between runs signals paging, not a healthy boundary.

Also know the honest limit. Routing experts to CPU trades VRAM for memory bandwidth, so fast dual-channel DDR5 matters more here than anywhere else in a local-AI build. Slow single-channel memory, or integrated graphics borrowing system RAM, can erase the gain entirely. This isn’t a free lunch — it’s you spending RAM bandwidth to buy back VRAM headroom.

Conclusion

Running a 235-billion-parameter model on a 16 GB card isn’t magic; it’s a property of MoE sparsity. Because only 22B of those parameters activate per token, llama.cpp lets you strand the bulky routed-expert weights in system RAM with --n-cpu-moe (a per-layer cutoff) or --override-tensor (a regex scalpel), while the small always-active attention and shared-expert tensors stay on the GPU.

The discipline is the same as any VRAM problem: match your offload to a measured budget, not a guess. Start conservative with all experts on CPU, reclaim them layer by layer while watching nvidia-smi, and add context last. Ignore the stale advice to pass --split-mode row — on a single GPU it’s irrelevant, and it’s deprecated anyway. Get the layer-vs-expert distinction right, confirm your -ot pattern with -v, and a model that once seemed to require a workstation will run on the GPU you already own.

If you’re still running into CUDA out of memory with dense models rather than MoE, see our companion guide to --n-gpu-layers, --mmap, and layer offloading, which covers the dense-model case these flags don’t help with.