Prompt caching
Getting the provider to remember the unchanging front of your prompt, so you stop paying full price and full latency for the same instructions.
See it
What it is
Every request makes the model re-read your entire prompt before it writes a word. If the first 8,000 tokens are the same system prompt, tool definitions, and few-shot examples every single time, you are paying full price and full latency to process identical text on repeat. Prompt caching lets the provider keep that processed prefix warm: cache hits typically bill at a fraction of input price and cut time to first token noticeably.
It only works on an exact prefix match, and that dictates prompt layout. Everything stable goes first (system prompt, tool schemas, the big reference document, examples), everything variable goes last (the user message, the timestamp, the retrieved chunks). One dynamic character near the top, like a 'today is' line or a shuffled ID, invalidates the whole thing.
Two gotchas worth budgeting for. Caches expire fast, often in minutes, so low-traffic endpoints may never hit; some providers also charge a premium to write the cache, which loses money if the prefix is used once. And there is a minimum cacheable length, so a 300-token prompt gets nothing. Read the usage numbers in the response to confirm you are actually hitting rather than assuming.
Ask AI for it
Restructure this LLM integration for prompt caching. Reorder the prompt so the stable prefix comes first (system prompt, tool definitions, reference docs, few-shot examples) and all variable content comes last (retrieved chunks, conversation turns, user input), removing anything dynamic from the prefix such as timestamps or random IDs. Add the provider's cache-control markers at the end of the stable block, then log cache-read versus cache-write token counts from the usage payload on every call and report the hit rate plus the cost delta so the win is measurable.