Temperature
The randomness dial on word choice. Low picks the safest next token every time; high lets unlikely ones through, for better or worse.
See it
What it is
At every step the model produces a probability for each possible next token. Temperature rescales that distribution before one is drawn. Near the bottom of the range the highest-probability token wins almost every time, so the output barely varies run to run, which reads as consistent and gets misread as safe. Push it up and the tail flattens, so unlikely tokens get a real chance, which reads as invention or as nonsense depending on the day. It changes which words get picked, never what the model knows.
The numbers do not port. Providers use different scales (some stop at 1, some go to 2), different defaults, and some endpoints ignore the parameter or refuse it outright, so any value you copy off a blog is a starting guess, not a setting. The shape of the advice does port: stay near the bottom of your model's supported range for extraction, classification, code, and anything you will parse; go higher for brainstorming, names, and copy variants where you want fifteen different swings; then pick the actual number by running task-specific evals at two or three settings. Top-p (nucleus sampling) does an overlapping job by cutting the tail off by probability mass, so tune one of the two and leave the other at its default rather than fighting yourself.
Gotcha: temperature 0 is not determinism. Batching on the server, floating-point addition not being associative on GPUs, and mixture-of-experts routing all mean the same prompt can return different text at 0. Pin a seed where the provider offers one, and write tests that assert on shape and meaning rather than an exact string. Second gotcha: low temperature makes wrong answers more repeatable, not more correct.
Ask AI for it
Audit and set sampling parameters per call site in this codebase instead of using one global default. First look up the supported range and default temperature for the model each call site uses, and skip the parameter where that endpoint does not accept it. Then choose from the low end of that range for any call that returns structured output, classifies, extracts fields, or generates code, and from the upper end for the copy-variant and brainstorming calls, requesting several completions there rather than reusing one. Leave top_p at its default so only one knob is moving. Confirm each choice by running the eval suite at two or three settings and keeping the winner rather than trusting a number from a blog post. Put the final value in a named constant next to the prompt with a comment saying which eval justified it, pass a fixed seed where the provider supports it, and make the eval suite assert on parsed fields and meaning rather than exact strings, since the lowest temperature still varies run to run.