guides/CONTEXT_LENGTH_FIX.md

The Critical Context Length Fix

This is the #1 issue people hit when deploying tool calling with VLLM.

The Problem

VLLM often defaults to context windows of 16K-32K tokens. This seems fine for normal chat, but tool calling needs significantly more context:

System prompt:                  3,000 - 5,000 tokens
Tool definitions (per tool):    500 - 2,000 tokens
  × 5-10 tools:                2,500 - 20,000 tokens
User message:                   100 - 1,000 tokens
Previous conversation:          1,000 - 10,000 tokens
Tool responses:                 2,000 - 20,000 tokens
Safety margin:                  5,000 tokens
─────────────────────────────────────────────────
Total needed:                   13,600 - 61,000 tokens

With a 16K context window, the model runs out of space mid-generation. The tool call gets silently truncated — you see incomplete JSON, missing arguments, or the model suddenly stops generating.

Symptoms

• Tool calls end mid-JSON: {"name": "get_weather", "arguments": {"loc
• Model stops generating after the first tool call in a multi-step workflow
• Second or third tool call in a conversation is always malformed
• Model "forgets" tool definitions and responds with plain text
• Works fine with 1-2 tools but fails with 5+

The Fix

# BEFORE (broken)
python -m vllm.entrypoints.openai.api_server \
  --model NousResearch/Hermes-3-Llama-3.1-70B-FP8 \
  --max-model-len 16384        # Default — too small

# AFTER (working)
python -m vllm.entrypoints.openai.api_server \
  --model NousResearch/Hermes-3-Llama-3.1-70B-FP8 \
  --max-model-len 131072       # 128K — full model support
  --max-num-seqs 4             # Reduce concurrency to fit KV cache
  --max-num-batched-tokens 132000
  --gpu-memory-utilization 0.90

Memory Math

The tradeoff is between context length and concurrent requests:

70B FP8 Model on 96GB GPU

Model weights (FP8):           ~40 GB
Available for KV cache:        ~46 GB (at 0.90 utilization)

KV cache per token per request:
  70B model ≈ 0.5 KB/token

Cost per concurrent request at 128K context:
  128,000 × 0.5 KB = 64 MB per layer × 80 layers ≈ 5 GB

Max concurrent requests:
  46 GB ÷ ~11.5 GB/request ≈ 4 requests

→ Use --max-num-seqs 4

12B FP8 Model on 96GB GPU

Model weights (FP8):           ~15 GB
Available for KV cache:        ~71 GB

→ Use --max-num-seqs 8 (or more)

Concurrency vs Context Tradeoff

Context Length	Max Seqs (70B)	Max Seqs (12B)	Tool Calling
16K	16	32+	Broken for multi-tool
32K	8	16	Marginal
64K	6	12	Good for simple workflows
128K	4	8	Reliable for complex workflows

Recommendation: Always use 128K. The reduced concurrency is worth it. If you need more throughput, use a smaller model rather than reducing context.

Why This Isn't Obvious

1. VLLM doesn't warn you when context is too small — it just generates truncated output
2. The default max-model-len varies by model and may not match the model's actual capability
3. Simple tests with 1-2 tools often pass even at 16K, so the issue only appears in production
4. The truncation looks like a model quality issue, not a configuration issue

Verification

After increasing context, verify with:

curl http://localhost:8000/v1/models | python -m json.tool

Check the max_model_len field in the response to confirm it's set to 131072.