Semantic caching
Reusing a previous answer when a new request means nearly the same thing, even though the words are different.
See it
What it is
Semantic caching reuses an earlier result when a new request means nearly the same thing, even if the wording differs. It embeds the incoming query, finds nearby cached queries, and returns a stored answer when the best match clears a chosen similarity threshold.
Reach for it on high-volume, repetitive, read-only questions where a small amount of answer reuse can save model cost and latency. Cache entries need more than a query vector: include the model, system prompt version, locale, data version, and any other input that can legitimately change the answer. GPTCache was the early open-source version of this pattern, and Redis ships a vector index that does the same job.
Gotcha: 'similar' is a product decision, not a universal number. A loose threshold can serve a cancellation-policy answer to a refund-policy question, while stale entries can outlive the source facts. Keep tenants and permission scopes separate, set a TTL, purge on source changes, and bypass the cache for personalized or state-changing requests.
Ask AI for it
Build a semantic response cache with Redis Stack and a RediSearch HNSW vector index using COSINE distance. Embed each normalized query with OpenAI text-embedding-3-small, partition keys by tenant, locale, model, system-prompt version, and knowledge-base version, and return a cached answer only below a threshold calibrated on labeled query pairs. Add a TTL, source-version invalidation, hit-rate metrics, and a hard bypass for personalized or state-changing requests.