Inference
Actually running a trained model to produce an answer. Training builds the model; inference is every live use of it afterward.
See it
What it is
Inference is one run through a trained model to get an output. Training changes the weights; inference holds those weights still, feeds in your tokens, and generates the next token repeatedly until the reply stops. A tool-using agent may perform dozens of inferences during one task, because every trip back to the model is another run.
Reach for the word when talking about the live side of an AI product: serving a model, sizing GPUs, measuring latency, or paying per token. Online inference handles requests as they arrive; batch inference groups many jobs together when an immediate answer does not matter.
Gotcha: model size is only the start of the bill. The weights and the growing KV cache must fit in accelerator memory, and latency splits into time to first token plus the pace of every token after it. A request can start quickly and still feel slow if generation crawls.
Ask AI for it
Serve Meta Llama 3.1 8B Instruct with vLLM's OpenAI-compatible /v1/chat/completions endpoint. Add a typed client that sends model, messages, temperature, max_tokens, and stream, then records time to first token, tokens per second, input tokens, output tokens, and the finish reason for every inference. Scrape queue time from vLLM's Prometheus /metrics endpoint. Support both streamed online requests and a bounded batch queue, reject work when the queue limit is reached, and return a clear timeout instead of retrying generation forever.