llama.cpp HTTP 400 “Context Size Exceeded”: Why –context-shift Won’t Save You and How to Fix It

The Symptom: A 400 Error That Shouldn’t Happen

You are running llama-server locally, pointed at a solid quantized model, and everything works — until a conversation grows long. Then, mid-session, the OpenAI-compatible /v1/chat/completions endpoint returns an HTTP 400 Bad Request with a terse body:

the request exceeds the available context size, try increasing it

Your first instinct is probably to blame the model or the memory. Neither is the problem. What actually happened is that your prompt plus conversation history outgrew the context window you reserved at launch, and llama-server responded exactly the way it was rewritten to respond: it rejected the request instead of silently guessing which of your messages to throw away.

This is one of the most frequently reported llamacpp-server behaviors in the local-AI community, and it confuses people because it used to behave differently. Older builds would quietly truncate the oldest tokens for you. Newer builds do not. Understanding that shift — and what the --ctx-size and --context-shift flags actually do — is the difference between a flaky setup and a reliable one.

What --ctx-size Actually Reserves

The flag --ctx-size N (also written -c N, and exposed as n_ctx in the API) sets the size of the key/value (KV) cache — the running buffer that holds the tokens your model can “see” at once. Every history token and every generated token has to fit inside it. A larger context window means the model can remember more of a conversation, but it also means more system RAM and VRAM is committed up front, because llama.cpp reserves the KV cache as a fixed allocation rather than growing it per-request.

The important mental model: --ctx-size is a hard ceiling, not a soft suggestion. When the tokenizer expands your messages array and the total lands at or above that ceiling, the server does not have room to run the prompt. On modern builds it rejects the request outright. You can confirm the number a model actually supports by checking the reported context length when it loads (often called n_ctx_train in logs), and you should never set --ctx-size above a model’s native trained length unless you know the model supports RoPE scaling to extend it.

If you routinely hit the wall, the first and simplest lever is to raise the ceiling — provided you have the RAM to back it. Launching with a larger --ctx-size commits more memory, so before you bump it, verify how much headroom you actually have.

What --context-shift Does (and Doesn’t) Do

The most common misconception is that --context-shift is the flag that makes the server “auto-manage” a long conversation. It is not. Context shift operates during the generation phase: when the KV cache fills up while the model is still emitting tokens, context shift discards the oldest tokens and slides the window forward so generation can continue instead of stalling. That is a real and useful capability, but it is scoped to the completion step.

It does not rescue an incoming request whose prompt is already too large. A request that exceeds --ctx-size before a single token is generated fails validation before context shift ever gets a chance to run. This is exactly the scenario in the widely-tracked llama.cpp issue #17284, where users launching with both --ctx-size 131072 and --context-shift still received the 400 because the request itself — not the generation — overflowed the window.

Part of the confusion is historical. An older code path automatically truncated an overflowing prompt server-side. That logic was removed because it had side effects and made the server’s behavior unpredictable. The result is the current, consistent contract: the server tells you the request is too big and lets you (or your client) decide what to do.

Why the Server Hands Truncation Back to You

The maintainer’s position on this is explicit and worth quoting directly: “context truncation is the job of the client, not the server.” The reasoning is sound. The server does not know which of your messages are safe to discard. Dropping the system prompt would change the model’s behavior. Dropping the most recent user turn would break the request. Blindly deleting oldest-first can silently corrupt the very thing you asked it to preserve. Handling that correctly requires understanding your application’s semantics, which is information the server — correctly — does not have.

This is why tools like Ollama and LM Studio appear to “handle” long conversations automatically: the truncation and compaction is implemented in their client layer, not in the inference engine underneath. If you are driving llama-server directly through its API, you are the client, which means the responsibility lands on your code.

A Working Fix, Step by Step

The reliable fix has two parts: give the model a context window that matches what you actually send, and stop sending more history than the window can hold.

1. Size the window to the real workload

Start with a ceiling that comfortably fits your longest plausible exchange. If your typical conversation is under a few thousand tokens, a --ctx-size of 8192 or 16384 is usually enough and keeps memory low. If you genuinely need long-context work, raise it deliberately and verify the memory cost.

2. Compress history on the client

Before assembling the messages array, run your own token estimate and trim. A practical, robust pattern is to keep the system prompt always, keep the most recent user and assistant turns, and summarize or drop the middle. Many local stacks maintain a rolling summary buffer for exactly this reason. You do not need a perfect tokenizer; a conservative character-to-token estimate (len(text) / 4) gets you close enough to stay under budget.

3. Catch the 400 and recover

Because the error is well-defined, your client can treat it as a signal rather than a crash. On receiving the “exceeds the available context size” response, drop the oldest non-essential turns, re-send, and log the event. This turns a hard failure into a self-healing loop.

4. Confirm before you scale up

If you do choose to raise --ctx-size, check that you have the memory to back the reservation. A window that is too large will exhaust RAM and send the operating system into swap, which punishes every request with latency that looks like a separate, more mysterious performance bug.

Common Traps to Avoid

• Assuming --context-shift fixes prompt overflow. It only applies during generation. An over-limit request still returns 400.
• Setting --ctx-size beyond the model’s trained length. This inflates memory without adding usable context, and can produce degraded output unless the model supports length extrapolation.
• Blindly truncating oldest-first. Dropping the system prompt or recent turns silently changes behavior. Compaction must preserve the parts that matter.
• Ignoring the memory cost. A bigger context window is a fixed RAM/VRAM commitment, not a free upgrade.

How to Verify You Fixed It

Reproduce the failure deliberately: craft a request you know exceeds your current --ctx-size, confirm the 400, then apply your client-side truncation and confirm the same request succeeds. Watch the server log line for the request path, which will show 400 for the failure and 200 once your trimming keeps the prompt inside the window. A healthy setup is one where the server never has to reject a request because your client never sends one it cannot fit.

The takeaway is not to fight the 400. It is a correctly implemented guardrail telling you your client and your context budget are out of sync. Align them, and the error disappears — along with the mystery of why a “perfectly good” model stops answering long conversations. For more on how the context window and KV cache interact with GPU memory, see my earlier breakdown of the KV cache spilling into system RAM, and for the authoritative server reference consult the llama.cpp server documentation.