Topic Modeling with LDA

Given ten thousand earnings-call transcripts, no analyst has time to read them all and manually tag each one with the themes it discusses. Topic modeling automates that: it looks at which words tend to co-occur across the whole collection and infers a small set of recurring "topics," then describes each document as a mixture of them — all without anyone providing labels.

The analogy: sorting a mixed bag of recipes by ingredient co-occurrence

Imagine a huge pile of recipe cards with no headings, just ingredient lists. You'd notice that "flour," "sugar," and "butter" tend to appear together, while "soy sauce," "ginger," and "scallion" tend to appear together in a different set of cards — and many cards use a bit of both, like a stir-fry with a side of bread. Without ever being told "this is a baking recipe" or "this is a stir-fry recipe," you could infer those two themes just from which words cluster together, and describe each card as a mix of themes rather than a single hard label. Latent Dirichlet Allocation (LDA) does exactly this with word co-occurrence across documents.

The generative story, one piece at a time

LDA assumes each document was generated by a simple process: first pick a mixture of topics for the document (say, 70% "margin pressure," 30% "supply chain"), then for each word in the document, pick a topic according to that mixture, and pick a word from whatever vocabulary distribution that topic favors. Two objects are learned from the data:

• $\phi_k$, the word distribution for topic $k$ — how likely each vocabulary word is under that topic. A "supply chain" topic gives high probability to words like "shipping," "backlog," and "logistics."
• $\theta_d$, the topic distribution for document $d$ — how much each topic contributes to that particular document.

In plain English: LDA doesn't know what a "topic" means in advance — it only knows that some words keep appearing together, and it works backward from those co-occurrence patterns to invent topic groupings and describe each document as a blend of them.

Worked example: three toy documents, two topics

Suppose a tiny corpus of three transcripts and a fitted two-topic model. Topic 1's top words: "backlog," "shipping," "suppliers." Topic 2's top words: "guidance," "raised," "beat." Document A is mostly about supply issues and gets $\theta_A = (0.85, 0.15)$ — 85% Topic 1, 15% Topic 2. Document B, an upbeat call, gets $\theta_B = (0.10, 0.90)$. Document C, which discusses both a supply hiccup and a guidance raise, gets $\theta_C = (0.45, 0.55)$ — genuinely mixed, which a single hard label could never capture but a soft mixture represents naturally.

Worked example: reading off a topic's signature

A fitted "margin pressure" topic might assign probabilities like: "costs" 0.08, "inflation" 0.06, "pricing" 0.05, "margin" 0.05, with a long tail of smaller weights on thousands of other words. Reading the top five or six words for each topic is usually how an analyst decides what to call the topic — LDA never names it "margin pressure" itself, it only produces the word-probability list; the human label is added afterward by inspection.

What this means in practice

Topic modeling is useful for exploring a large unlabeled corpus before deciding what to build a supervised classifier for — it can reveal that "supply chain concerns" was quietly a bigger recurring theme across a sector's earnings calls than anyone expected, without a single manual read-through. The number of topics $K$ has to be chosen in advance and isn't learned from the data, and the resulting topics need a human to inspect the top words and assign a sensible name; LDA finds the statistical groupings, not the English label for them.