Turning LLM Outputs Into Model Features

An LLM reads a company's risk-factors section and writes three paragraphs summarizing the main concerns. That's useful for a human analyst to read, but it's not something a regression, a gradient-boosted tree, or a factor model can consume directly — those models need a number, not a paragraph. Turning an LLM's qualitative read of a document into a numeric feature is the bridge between "the model can understand text" and "the text can enter a quantitative pipeline," and how carefully that bridge is built determines whether the resulting feature is a genuine signal or a source of noise dressed up as data.

Two ways to get a number out

The direct approach asks the LLM to output a number itself — "rate the severity of supply-chain risk mentioned in this filing on a scale of 1 to 5" — constrained to a fixed schema (see schema-constrained JSON output) so the response is reliably parseable as {"severity": 3} rather than a sentence that has to be parsed by hand. This is simple but opaque: the model's internal reasoning for choosing 3 over 4 isn't inspectable, and the same filing read twice can produce different numbers if the model's output isn't deterministic.

The indirect approach asks the LLM to extract structured, checkable facts — "does the filing mention a specific new competitor by name? does it mention a specific dollar figure for a legal contingency?" — as booleans or extracted values, and computes the numeric feature afterward with ordinary code from those extracted facts. This is more work upfront but produces a feature whose provenance is traceable: you can always go back and check exactly which sentence produced a "yes."

Worked example: a risk-severity feature from 500 filings

Suppose you want a numeric "litigation risk intensity" feature across 500 10-Ks. The direct approach prompts the LLM once per filing: "on a scale of 0–10, how severe is the litigation risk disclosed here?" Run twice on the same filing, one company might score 4 the first time and 6 the second, purely from model variance — noise that looks like signal to a model trained on it but degrades out-of-sample performance. The indirect approach instead extracts concrete facts — number of legal proceedings mentioned, whether a dollar liability figure is disclosed, whether "material adverse effect" appears — and combines them with a fixed formula, e.g. $2 \times (\text{proceedings count}) + 3 \times (\text{dollar figure disclosed}) + 5 \times (\text{phrase present})$. Rerun, those extracted facts are far more stable than a free numeric judgment, because "does this filing mention a dollar figure" is a much narrower question than "rate the severity."

What this means in practice

Any LLM-derived feature entering a quantitative model should be treated the way any other engineered feature is treated — checked for stability across reruns, checked for leakage (was the LLM run point-in-time, without seeing information from after the feature's date), and checked for whether it actually adds predictive power over simpler baselines already in the model. It's easy for an LLM feature to feel sophisticated and add nothing, or worse, to add noise that a backtest doesn't distinguish from signal until it's live.