Decoding Strategies and Temperature

At every step, a language model does not directly output a word — it outputs a probability distribution over every possible next word. Decoding is the separate step of turning that distribution into an actual choice, and different decoding strategies produce noticeably different text from the exact same model.

Greedy decoding always picks the single highest-probability word at each step — deterministic, but prone to bland, repetitive output because it never risks a slightly-lower-probability word that might lead somewhere better. Sampling instead draws a word randomly according to the model's probabilities, adding variety at the cost of occasional weird or incoherent choices. Temperature is a knob applied before sampling that reshapes the distribution: a temperature below 1 sharpens it toward the most likely words (approaching greedy as temperature goes to zero), while a temperature above 1 flattens it, making unlikely words relatively more probable and the output more random.

Worked example. Suppose the model assigns probabilities 0.7, 0.2, and 0.1 to three candidate next words. At temperature 0.5, those probabilities sharpen to roughly 0.85, 0.12, and 0.03 — the model becomes more confident and predictable. At temperature 2.0, they flatten toward roughly 0.45, 0.32, and 0.23 — the three choices become nearly as likely as each other, producing more varied but less focused output.

For finance applications like generating draft commentary or summarizing filings, a low temperature is usually preferred, since factual consistency matters more than creative variety.