Large Language Models

A large language model does one thing: given a stretch of text, it produces a probability for every possible next token. Nothing else. Everything it appears to do — summarise a 10-K, pull the guidance number out of a transcript, explain a trade — falls out of that one objective applied at enormous scale.

The useful mental image is an extraordinarily well-read autocomplete. Your phone's keyboard predicts the next word from the last two; an LLM predicts it from the last hundred thousand, having read a substantial fraction of the public internet. At small scale that gets you plausible-sounding gibberish. Past a certain size and training volume, the only way to keep getting better at the prediction task is to actually track who did what to whom, what a number means in context, and how an argument is structured. Capability arrives as a by-product of compression.

The generation loop

Text is first broken into tokens, roughly word-fragments: "unprofitable" might be three of them. The model runs a stack of transformer layers over the sequence, using The Self-Attention Mechanism so that every position can look back at every earlier position, and outputs a raw score, or logit, for each token in the vocabulary. Those scores go through a softmax to become probabilities, one token is chosen, it gets appended to the input, and the whole thing runs again.

Two consequences follow immediately. Generation is inherently sequential, so producing 500 tokens takes 500 forward passes — that is why latency scales with output length, not input length. And each step depends on everything already written, so one bad token early poisons everything after it.

Worked example: what temperature does

Suppose at some step the model's logits for four candidate tokens are $3.0$, $2.0$, $1.0$ and $0.0$. Temperature $T$ divides every logit before the softmax.

At $T = 1$ (no change), exponentiating gives $20.086$, $7.389$, $2.718$, $1.000$, summing to $31.193$:

$$P = (0.644,\ 0.237,\ 0.087,\ 0.032)$$

At $T = 0.5$ the logits become $6, 4, 2, 0$, exponentiating to $403.43$, $54.60$, $7.39$, $1.00$, summing to $466.42$:

$$P = (0.865,\ 0.117,\ 0.016,\ 0.002)$$

At $T = 2$ the logits become $1.5, 1.0, 0.5, 0.0$, exponentiating to $4.482$, $2.718$, $1.649$, $1.000$, summing to $9.849$:

$$P = (0.455,\ 0.276,\ 0.167,\ 0.102)$$

In plain English: low temperature sharpens the model's existing preference, high temperature flattens it. For extraction tasks — pull the revenue figure, classify this sentence — you want $T$ near zero so the output is near-deterministic and reproducible. Creative rewriting is the only place a high temperature earns its keep, and in a research pipeline that is almost never what you want. See Decoding Strategies and Temperature.

Worked example: perplexity

Quality of a language model is usually reported as perplexity, which answers "on average, how many equally likely options was the model choosing between?"

Take a four-token sequence where the model assigned the true tokens probabilities $0.5$, $0.25$, $0.1$ and $0.8$. Average the negative log probabilities:

$$-\tfrac{1}{4}\left[\ln 0.5 + \ln 0.25 + \ln 0.1 + \ln 0.8\right] = -\tfrac{1}{4}(-4.605) = 1.151$$

Then exponentiate:

$$\text{perplexity} = e^{1.151} = 3.16$$

The model was about as uncertain as someone guessing uniformly among three options at each step. Lower is better; a perplexity of 1 would mean perfect prediction. It is a useful sanity check when adapting a model to financial text, because a general model's perplexity on dense filing language is usually much worse than on news.

Where they help, and where they wreck a backtest

The reliable wins are reading tasks at volume: turning a transcript into structured fields, tagging a filing's risk factors, deduplicating a news feed, drafting the first version of a research note. These are tasks where a human could verify the output quickly and where volume, not judgement, is the bottleneck.

The unreliable use is asking the model to predict returns directly. It has no price data in its weights, only text about prices, and it will produce a confident answer anyway.

The practical discipline is to treat the model as one stage in a pipeline rather than the pipeline itself. Give it a narrow, checkable job; log the exact prompt, model version and output for every call so a result can be reproduced months later; and hold back a hand-labelled sample to measure extraction accuracy the same way you would validate any other data vendor. An LLM that silently changes behaviour between releases is a data-quality problem wearing a clever disguise.