Why Ollama Crashes on the Second Request: OLLAMA_NUM_PARALLEL, MAX_QUEUE, and the KV Cache Multiplier

You run ollama run llama3.1:8b, ask a question, and it answers fine. Then a teammate — or a second browser tab, or an agent you wired up — fires a request at the same model, and suddenly the whole thing dies with CUDA out of memory or model requires more system memory than is available. The model worked three seconds ago. Nothing changed. So what actually happened?

The answer is that Ollama began running inference concurrently by default, and every extra concurrent request doesn’t just add a little overhead — it duplicates the KV cache, the single most memory-hungry part of a running model. Once you understand the three environment variables that govern this behavior, the crashes become predictable and preventable.

The Symptom: One Request Works, Two Crash

The classic reproduction looks like this. You have a 16 GB GPU. You load a 13B–14B model quantized to Q4, which leaves you comfortable but not luxurious headroom — say 2–3 GB of free VRAM after the weights and a full context window are loaded. A single request runs at a healthy token rate. Then two requests arrive close together, and the process aborts.

The giveaway is in when the crash happens. It’s not at model load — the model loads fine on its own. It’s precisely when a second concurrent request forces the runner to allocate a second context window. If your logs show a clean start, a first successful generation, and then an OOM abort, concurrency is the first thing to suspect, not your model choice or your quantization.

This is distinct from the more common “my model is too big” OOM, which fails at load time before any token is produced. A concurrency-triggered OOM fails mid-session, after at least one good generation — which is exactly why it feels so confusing.

The Mechanism: Why Parallel Requests Multiply VRAM

When a transformer model generates tokens, its memory footprint splits into two parts. The model weights are loaded once and shared by every request. The KV (key-value) cache, however, holds the attention state for a specific sequence, and it grows with both context length and batch size. Every distinct request needs its own KV cache.

This is the multiplier. If your KV cache for a single 32k context request occupies, say, 2 GB, then two parallel requests need roughly 4 GB, four need 8 GB, and so on. The weights stay the same, but the variable part of your memory usage scales nearly linearly with the number of concurrent requests. On a GPU that was already 90% allocated to weights plus one context, a second context is all it takes to tip you over the edge.

This is also why the same model feels stable in single-user desktop use but explodes the moment you expose it through an API gateway, a local chat UI with multiple sessions, or several automation agents that all happen to call it at once. The model didn’t change — the number of simultaneous sequences did.

Diagnosing the Problem Before It Happens

You can confirm the diagnosis without waiting for a crash. Ollama exposes a /api/ps endpoint (and the ollama ps command) that reports exactly what is loaded and how much memory each model is using. Before you wire Ollama into anything that fires concurrent requests, check the baseline:

ollama ps
# NAME               ID              SIZE      PROCESSOR    UNTIL
# llama3.1:8b        xxxxx           6.1 GB    100% GPU     4 minutes from now

Then start two requests at once (a second terminal with a second ollama run, or two curl calls against /api/generate), and watch the memory climb. On Linux, nvidia-smi in a third terminal shows the exact moment VRAM tops out. If you see the resident usage roughly double and then the process die, you’ve confirmed it.

For headless servers, the two environment variables that matter are set in the environment of the ollama serve process — not in the client that calls it. A very common mistake is setting OLLAMA_NUM_PARALLEL=1 in the shell that runs ollama run; that does nothing, because the constraint lives on the server side, applied per model when requests are admitted.

The Three Environment Variables That Control Concurrency

OLLAMA_NUM_PARALLEL — how many requests run at once

This sets the maximum number of requests a single model will process simultaneously. The default auto-selects either 4 or 1 based on available memory when the model loads. For a single-user desktop, a larger value ages gracefully; for a small GPU serving multiple agents, setting it explicitly to 1 forces requests to serialize instead of stacking multiple KV caches.

# serialize requests to one model on a VRAM-constrained GPU
export OLLAMA_NUM_PARALLEL=1
ollama serve

OLLAMA_MAX_QUEUE — how many requests wait in line

When concurrent requests exceed OLLAMA_NUM_PARALLEL, the excess is not dropped — it is placed in a FIFO queue. OLLAMA_MAX_QUEUE caps how many requests may sit waiting before Ollama rejects the next one outright. The default is generous (512). Lowering it doesn’t save VRAM, but it constrains how deep a backlog of pending, memory-reserving requests can grow, and it lets your callers fail fast with a clear error instead of hanging.

# reject instead of queue above 16 pending requests
export OLLAMA_MAX_QUEUE=16

OLLAMA_KEEP_ALIVE — how long a loaded model lingers

After a request finishes, the model stays resident in memory for a configurable window (default 5 minutes) so the next request doesn’t pay a full reload. In a bursty automation environment, several recently-used models can all sit loaded at once, invisibly consuming VRAM that you think is free. Setting OLLAMA_KEEP_ALIVE to a short value (or 0 to unload immediately) frees that memory, but at the cost of a reload penalty on the next call. It’s a direct tradeoff between latency and headroom.

# unload immediately after idle — reclaim VRAM fast
export OLLAMA_KEEP_ALIVE=0

A fourth variable, OLLAMA_MAX_LOADED_MODELS, deserves a mention. Its default (0) lets Ollama keep up to three models resident per GPU (or three on CPU-only machines). If you’re swapping between a large and small model frequently, that’s convenient; if every megabyte matters, capping it to 1 forces older models to unload before a new one loads.

Common Pitfalls That Make It Worse

• Setting the variables in the client, not the server. The concurrency limits are read by the ollama serve process. If you run Ollama as a system service (systemd on Linux, LaunchAgent on macOS, a Windows service), the environment variables must be injected into that service’s environment — editing the shell that launches ollama run has no effect.
• Confusing KV cache quantization with concurrency. Quantizing the cache (OLLAMA_KV_CACHE_TYPE=q8_0) and enabling flash attention (OLLAMA_FLASH_ATTENTION=1) shrink each cache, which is valuable — but they do not stop the multiplication when parallel requests arrive. Both fixes are worth applying, but they address a different axis of the problem.
• Treating the default NUM_PARALLEL as harmless. The auto-selected default can be 4, meaning four simultaneous KV caches. On a GPU that’s comfortable with one but not four, the crash looks arbitrary because it depends on whether requests overlap in time — which is precisely what makes it intermittent and hard to reproduce.

Verifying the Fix

After setting the variables and restarting the server, reproduce the original failure deliberately. Fire two requests at once and confirm the second is queued (not run in parallel) when NUM_PARALLEL=1 — you should see a single steady generation followed by the queued one, with no memory spike above the single-context baseline in nvidia-smi. Then saturate MAX_QUEUE deliberately and confirm the next request is rejected with a clear error rather than hanging or crashing.

The formal reference is the Ollama FAQ, which documents OLLAMA_NUM_PARALLEL, OLLAMA_MAX_QUEUE, and OLLAMA_MAX_LOADED_MODELS along with their default behavior. For the deeper mechanics of how requests are admitted and queued, the analysis in Rost Glukhov’s write-up on Ollama parallel request handling is a clear, correct reference.

Conclusion

The single most counterintuitive thing about running a local model is that concurrency, not model size, is often what kills it. A model that runs perfectly for one user will OOM the moment a second sequence arrives, because each parallel request clones the KV cache while the weights are shared. The fix is rarely “buy more VRAM” — it’s usually “control how many requests run at once.” Set OLLAMA_NUM_PARALLEL to match your real headroom, cap OLLAMA_MAX_QUEUE so backlog fails fast instead of festering, and be deliberate about OLLAMA_KEEP_ALIVE and OLLAMA_MAX_LOADED_MODELS on multi-agent or automation-heavy boxes. Diagnose with ollama ps and nvidia-smi, reproduce deliberately, and the once-mysterious crash becomes a predictable, documented boundary.

If you’re hitting the more general case of a CUDA out-of-memory on a 16 GB card, see our companion guide on diagnosing llama.cpp CUDA out-of-memory on a 16 GB card.