Context window
The maximum tokens a model can process at once, prompt plus reply. Not a memory: a desk. Overflow is an error, not a polite trim.
See it
What it is
The context window is the ceiling on how many tokens a model can process for one invocation. System prompt, chat history, retrieved chunks, tool definitions and the reply it still has to write all compete for the same budget, and exactly how input and output are counted against it varies by provider, so read the docs for the model you actually picked. Go over and the usual answer is a hard error, not a helpful trim: nothing quietly drops your oldest messages for you. Any truncation is a policy your code or your framework chose. Windows run from a few thousand tokens on small local models to a million on frontier ones, at roughly 4 characters of English per token.
It drives architecture more than any other number here. If your docs do not fit, you need retrieval instead of pasting. If chats run long, you need a rolling summary. Reserve room for the answer too: a request that fits only because you forgot the model still has to write 2,000 tokens back is a request that fails. And if cost matters, you trim before you send, because you pay for every input token on every single turn.
Gotcha: fitting is not the same as using. Recall sags in the middle of a long window (the 'lost in the middle' effect) and quality degrades as the window fills with stale, half-relevant text, which people call context rot. A tight 8k prompt of exactly the right material routinely beats a 400k dump of everything you own.
Ask AI for it
Rebuild this chat feature to live inside an explicit context budget. Read the selected model's context limit from config, reserve the configured maximum output tokens plus a safety margin, and enforce inputBudget = contextLimit - outputReserve - margin before every call, counting with that model's own tokenizer. Assemble each request in priority order: system prompt, the 5 most relevant retrieved chunks, a running summary of older turns, then the most recent messages verbatim. Apply one explicit truncation policy (drop the oldest middle turns into the summary, never the instructions), record what was dropped, and fail loudly rather than sending a request you already know is over budget. Log the token count per request and warn once usage passes 80 percent of the input budget.