Top-p / nucleus sampling
A word-choice cutoff: keep only the smallest group of likely next tokens whose probabilities add up to p, then sample from that group.
See it
What it is
Top-p, also called nucleus sampling, sorts the possible next tokens by probability and keeps the smallest group whose combined probability reaches p. The model samples only from that group. At 1 the full tail remains eligible; lower values cut away more unlikely choices at every step, so the size of the candidate pool adapts to how certain the model is. It comes from the 2019 paper 'The Curious Case of Neural Text Degeneration' by Ari Holtzman and colleagues, which proposed it as a cure for the flat, looping text that likelihood-maximising decoders produced.
Reach for it when you want sampling but need a direct way to exclude the long tail of odd word choices. It overlaps with temperature, which reshapes the whole probability distribution instead of drawing a probability-mass boundary. Tune one against an eval set and leave the other at its default so you can tell which change helped.
Gotcha: top-p does nothing when decoding is greedy. In Hugging Face Transformers, for example, `top_p` needs `do_sample=True`. A lower value also does not make an answer more factual. It only narrows which next tokens can be drawn, so a confidently wrong continuation can sail straight through.
Ask AI for it
Add nucleus sampling to this Hugging Face Transformers generation call. Use `generate()` with `do_sample=True`, expose `top_p` as a validated configuration value greater than 0 and at most 1, and leave `temperature` at its existing default while top-p is being evaluated. Run the same fixed prompt set at top-p values 0.8, 0.9, 0.95, and 1.0 with at least 20 seeded generations per value. Report task accuracy, format failures, and distinct outputs, then keep the lowest-cost setting that passes the existing quality threshold.