Batch inference
Running many independent model requests as a queued job, trading instant answers for lower cost or higher throughput.
See it
What it is
Batch inference processes many independent model requests as a queued job instead of answering each one interactively. The provider or serving system can schedule the work efficiently, trading immediate responses for lower cost, higher throughput, or both. The OpenAI Batch API is the standard shape: upload a JSONL file of requests, results land within 24 hours, tokens bill at roughly half the interactive price.
Reach for it when results can arrive later: classifying a catalog, embedding a document collection, backfilling summaries, or running an eval suite. Give every request a stable ID, make submission idempotent, and write results incrementally so one bad record does not waste completed work.
Gotcha: batching requests does not mean stuffing many records into one huge prompt. Keep their contexts separate or they can leak into each other and become impossible to retry cleanly. Results may arrive out of order, jobs can partially fail, and provider queues have file and time limits, so reconcile by ID and retry only failed items.
Ask AI for it
Build an OpenAI Batch API worker for /v1/chat/completions. Write one JSONL request per record with a unique custom_id, method, url, and body; upload the file, create the batch through /v1/batches, poll its status with exponential backoff, and stream the output file into durable storage. Reconcile results by custom_id, record per-item errors, make submission idempotent, and retry only failed records in a new batch.