Context overflow; truncation
When a request is too large for the model's token window. Overflow is the failure; truncation is your rule for dropping content to fit.
See it
What it is
Context overflow is what happens when the assembled input plus the space reserved for output exceeds the model's context window. Most raw model APIs reject that request outright, the way OpenAI returns a `context_length_exceeded` error. Truncation is the policy that removes or shortens material before the call so it fits. If old chat turns quietly disappear, your application or framework chose that policy, not the model.
Reach for deliberate truncation in long chats, retrieval pipelines, and document tools where the available material can outgrow a fixed budget. Count with the selected model's tokenizer, reserve output space first, then keep material by priority: durable instructions, the current user request, the most relevant evidence, and recent turns. Summarise or drop the lowest-priority remainder and record what changed.
Gotcha: slicing the oldest text is simple and often wrong. It can remove a decision the current turn depends on while leaving pages of greetings, or cut through JSON, code, and multi-token delimiters. Truncate whole semantic units, never characters, and fail loudly if the non-negotiable instructions and current input cannot fit by themselves.
Ask AI for it
Add explicit context truncation before every model call. Use `tiktoken.encoding_for_model()` to count the assembled request, subtract the configured `max_output_tokens` and a 5 percent safety margin from the model's context limit, then fit whole blocks in this order: system prompt, current user message, retrieved chunks by reranker score, recent turns, and the running summary. Never slice raw characters or split a message. Log every omitted block with its token count, and throw a typed error if the system prompt plus current message alone exceed the input budget.