Bag of Words and TF-IDF

A model can't read a 10-K filing directly — it needs numbers. The oldest and still most common fix is to throw away word order entirely and just count which words show up. It sounds too crude to work, but for tasks like flagging a filing's tone or classifying a news headline, raw word counts, properly reweighted, remain a strong and cheap baseline.

The analogy: judging a room by what's shouted, not the sentences

Imagine walking past a room full of conversations and, without hearing full sentences, just tallying which words get shouted most often. You'd lose all grammar and order, but a tally full of "lawsuit," "restated," and "material weakness" tells you something is wrong even without hearing a single complete sentence. Bag of words is that tally: it discards sentence structure and keeps only word frequency.

Building the bag, then re-weighting it

Bag of words represents a document as a vector of word counts — one entry per word in the vocabulary, ignoring order entirely. Two documents with the same words in different orders produce the same vector.

The problem: words like "the" and "company" appear in nearly every filing, so raw counts let uninformative common words dominate the vector. TF-IDF (term frequency–inverse document frequency) fixes this by down-weighting words that appear in many documents:

$$\text{tfidf}(w, d) = tf(w, d) \times \log\!\left(\frac{N}{df(w)}\right)$$

Here $tf(w,d)$ is how often word $w$ appears in document $d$, $N$ is the total number of documents in the collection, and $df(w)$ is the number of documents that contain $w$ at all. In plain English: a word gets a high score in a document if it appears often there but rarely elsewhere — exactly the property that makes a word distinctive rather than generic.

Worked example: two filings, one word

Suppose a collection of $N = 1{,}000$ annual reports. The word "restatement" appears in only $df = 5$ of them, but 8 times in one particular filing's risk section ($tf = 8$). Its inverse-document-frequency weight is $\log(1000/5) = \log(200) \approx 5.3$, so $\text{tfidf} = 8 \times 5.3 \approx 42.4$ — a large, attention-grabbing score. Compare "company," which appears in all 1,000 filings ($df=1000$) and 20 times in the same document ($tf=20$): its weight is $\log(1000/1000) = \log(1) = 0$, so $\text{tfidf} = 20 \times 0 = 0$. Despite appearing far more often, "company" contributes nothing to the vector, because it carries no distinguishing information — exactly the reweighting bag-of-words counts alone would miss.

Worked example: comparing two short headlines

"Company beats earnings estimates" and "Company misses earnings estimates" share three of four words. A pure bag-of-words model built on unigrams (single words) sees these as nearly identical vectors, since "beats" and "misses" are each just one word among four — a real weakness, since the sentiment is opposite. Adding bigrams (word pairs) as extra vocabulary entries, like "beats earnings" versus "misses earnings," lets the model pick up some of the local context that single-word counts throw away, at the cost of a much larger vocabulary.

What this means in practice

TF-IDF vectors remain a fast, interpretable baseline for classifying filings, headlines, or transcripts before reaching for embeddings or a language model — every dimension corresponds to an actual word, so you can inspect exactly which terms drove a classification. The tradeoff is that word order and context are gone: negation ("not profitable"), sarcasm, and multi-word idioms are all invisible to a plain bag of words, which is precisely the gap that word embeddings and later transformer-based models were built to close.