The claim worth testing
A 27B model on an 8GB graphics card. Every guide I found said you need 24GB for this class of model, and the quantization charts back that up at first glance. But look closer at the IQ4_XS row and the picture changes: roughly 6% quality loss against the base model, with the file size dropping to about 14GB.
14GB does not fit in 8GB. So the honest question That is what this post is about, because the benchmarks do not answer it.
What the settings actually were
If you want to reproduce this, the configuration matters more than the model choice.
**Quantization:** Unsloth IQ4_XS. Standard practice for a small card.
**KV cache:** Quantized to 4-bit. This is not optional at this size. Without it your context window eats the VRAM you needed for the model itself. There is a trap here worth knowing about: quantizing the KV cache can silently push work onto the CPU, and I walked through why that made prefill 30x slower in llama.cpp rather than assuming it helped.
**Context:** 70K tokens. Deliberately modest. If you are running at 8GB you should be picking the smallest context that gets the job done, not the largest one that fits.
**Offload:** Around 26 layers sent to the GPU. The rest lives in system RAM. Total VRAM landed around 7GB, which leaves room for the operating system, and you will need that room. If you want to understand what the offload flags are actually doing, ComfyUI’s –lowvram vs –novram vs –normalvram comparison covers the same mechanics in a different runtime.
**CPU threads:** Set to match the physical core count. Six cores, six threads. Setting this to your logical core count is a common mistake and it makes things worse, not better.
**Speculative decoding:** MTP mode with 2 max draft tokens. If this is unfamiliar, the HuggingFace writeup on assisted generation explains the mechanism.
**Sampling:** Temperature 1, top-k 20, repeat penalty 1, min-p 0. These come from the quantization author’s recommendation, which is usually a better starting point than whatever the model card says.
The two things that will waste your afternoon
This is the part the videos skip, and both cost real time.
**First: disable the HTTP timeout in your harness.** The generation speed on this setup is genuinely slow, slow enough that the timeout fires and kills the tool call before the model finishes writing. If you see the model apparently stuck in a loop when it is trying to write output, this is what is happening. It is not looping. It is being cut off.
**Second: raise your max output tokens.** For the same reason, writing a large file will trip the default limit and the process gets terminated mid-write. The default exists to protect you from runaway generation on fast hardware. On an 8GB card it just breaks your work.
One more setting matters here, and it is not a bug.
**Remove the reasoning budget, and keep thinking enabled.** This sounds contradictory and it is not. The model uses an enormous number of tokens thinking. One moderately sized prompt consumed about 20,000 tokens just on reasoning before producing any output. If you cap that, the model gets cut off mid-thought and the output quality collapses. On slow hardware the thinking is expensive, but it is the thinking that makes the model good. Cutting it to save time defeats the point of running the model at all.
What the numbers looked like
Four tasks, all completed in a single shot with no agent looping.
A Minecraft 3D landing page took two and a half hours. Average throughput was 4.2 tokens per second, starting near 6 and degrading as context grew. The same prompt through a frontier model produced a comparable result.
A cyberpunk neon city 3D scene took about an hour and a half at roughly 5 tokens per second.
A detailed personal website prompt took two hours at 4.6 tokens per second.
An interior design company landing page took an hour and a half at 5 tokens per second.
The degradation is the interesting part. You do not get a steady 5 tokens per second. You get 6 at the start and 4 by the end, and the decline tracks context length. That is the cost of KV cache growth on a card that has no room for it, and it is why long chats slow to a crawl on setups that looked healthy at the start.
Is this actually useful
Honest answer: it depends what you want.
If you need an interactive assistant, 4 tokens per second will drive you up the wall. You will wait on every response.
If you need something to grind on a well-specified task while you do something else, it works fine. Two and a half hours for a landing page is not a workflow. It is a background job. Treat it that way and the numbers stop being disappointing.
Five hours of the afternoon went into the two problems described above. Almost none of it was model tuning. The two failures that cost the most time were a timeout setting and a token limit. Neither is a machine learning problem. Both are the kind of thing you only find out about by actually running the thing, which is exactly why a working configuration and an honest account of what broke is more valuable than another benchmark table.
The transferable part
If you take one thing from this, take the shape of the problem rather than the specific numbers.
Small cards fail in a predictable order. First you run out of VRAM for the model, so you quantize. When the KV cache then becomes the problem, the fix is often counterintuitive. Then your context eats the space you saved, so you quantize the cache. Then you offload, and pay a bandwidth tax on every single token. Then the slowdown trips timeout and size limits you did not know were configured. Every one of those steps is where the real time goes, and none of them show up in a comparison chart.
The settings in this post were tuned for an 8GB card. On a 16GB card the same reasoning applies with different constants. The failure order is the same. The limits are just further out.
A note on where these numbers came from
I did not run this test. The configuration and timings come from a video walkthrough that tested Qwen 3.8 27B on a single RTX 4060, and I am relaying it here because the settings and the failure modes line up exactly with what I hit running a 14B video model on a 16GB card.
That is the reason this belongs on the site. The specific card differs, the model class differs, and the workload differs. The sequence of failures does not. You run out of memory, you quantize, the cache becomes the constraint, you offload, and then the slowdown breaks tooling that was never designed for how slow things got.
The two fixes that mattered most, disabling an HTTP timeout and raising an output token limit, are both the kind of thing you find by watching the process die rather than by reading documentation. That pattern has held for every local AI setup I have touched, and it is why I keep writing about the failures instead of the benchmark scores.
Leave a Reply