When Is a Liability Not a Liability? The Dictionary That Fixed Financial Text Analysis

After Tetlock showed that counting negative words in financial text predicted market behaviour, everyone piled in. Textual analysis was the new frontier. Researchers ran word-count dictionaries over annual reports, earnings calls, analyst notes and news.

And nearly all of them used the same tool: the Harvard psychosocial dictionary, a word list developed by researchers in psychology and content analysis to classify language into categories like positive, negative, strong and weak.

Tim Loughran and Bill McDonald asked the question that in retrospect is glaringly obvious. Is a dictionary built by psychologists the right tool for reading a company's annual report?

They went and checked, and the answer was a fairly emphatic no.

The problem: borrowing a ruler calibrated for something else

Here is the failure, and it is delicious.

The Harvard dictionary flags a word as negative if it is negative in ordinary English. Reasonable enough. Now open a 10-K, the annual report every US public company must file, and look at the words the dictionary flags:

• Liability. Negative in English. In a 10-K, it is one half of the balance sheet. Every company has liabilities. Having them is not bad news, it is accounting.
• Cost. Negative in English. In a 10-K, it is the cost of goods sold. It appears constantly, in every filing, forever.
• Tax. Negative in English. In a 10-K, it is a line item.
• Capital. Flagged as negative in the Harvard scheme. In finance, it is the substance of the whole business.
• Vice. Negative in English. In a 10-K, it usually appears in the phrase "vice president."
• Crude. Negative in English. In an energy company's filing, it is oil.
• Mine. Negative in the dictionary's classification. For a mining company, it is what they do.

Loughran and McDonald quantified the damage. Examining a large sample of 10-K filings over 1994 to 2008, they found that almost three quarters of the words the Harvard dictionary flagged as negative are not negative in a financial context. Three out of four.

Think about what that means for every study that used this tool. The "negativity score" of a filing was, to a substantial degree, measuring how much the company talked about its liabilities, its costs and its taxes. Which is to say, it was measuring how big and how complicated the company is, and possibly what industry it is in, dressed up as a measurement of tone.

That is not a small calibration error. It is a systematically biased instrument, and the bias correlates with exactly the firm characteristics (size, industry, complexity) that also predict returns. So a researcher could easily find that "negative tone predicts returns" when what they had really found is that "being an oil company predicts returns."

The key idea via analogy: words have a job, and the job depends on the room

A word does not have a fixed emotional charge. It has a charge in context.

"Cancer" is a devastating word in a medical report and a neutral one in an oncology company's product description. "Liability" is a worry in a conversation and a line item in a balance sheet. Language is domain-specific, and if you want to measure the tone of a specialised document, you need a dictionary built for that domain.

So Loughran and McDonald built one. They went through the vocabulary that actually appears in 10-K filings and hand-classified it by what the word means to a financial reader, producing a set of word lists:

• Negative words that really are negative in a financial context.
• Positive words, with the same care.
• Uncertainty words, which capture hedging and ambiguity, a distinctly financial concept that general dictionaries have no category for.
• Litigious words, signalling legal exposure.
• Strong and weak modal words, which capture how confidently management is speaking. "Will" is a different signal from "might."

The last two categories are the ones that show they were thinking as finance people rather than linguists. Uncertainty and litigiousness are not emotions. They are financial states, and they matter enormously to an investor. No general-purpose dictionary would ever have thought to build them.

Then they showed the new tool worked. Their finance-specific word lists were related to a whole battery of outcomes that matter: the market's return around the 10-K filing date, trading volume, subsequent return volatility, the incidence of fraud, the disclosure of material weaknesses in internal controls, and unexpected earnings. The tone of a filing, measured properly, carries real information about the company's future.

Why it mattered

• It fixed the foundations of a fast-growing field. Textual analysis in finance was expanding rapidly on top of a broken instrument. This paper caught the error before it had been baked into a decade of research, and provided the replacement.
• The word lists became the standard. The Loughran-McDonald dictionaries are freely available and are, to this day, the default tool for financial text analysis in both academia and industry. Very few papers achieve that kind of infrastructural status.
• It made a general point about domain adaptation that has only got more important. Every off-the-shelf language tool carries the assumptions of the corpus it was trained on. That was true of the Harvard dictionary in 2011 and it is true of general-purpose language models today. A model trained on the internet has learned the internet's associations for the word "exposure," and those are not a credit analyst's associations. Financial language is a dialect, and tools must be calibrated to it.
• It gave textual analysis credibility. By demonstrating that a properly-built measure of tone predicts filing-date returns, volatility, fraud and earnings, the paper established that there is real signal in corporate text, and that it was worth doing well.
• It introduced categories nobody had thought to measure. Uncertainty and litigiousness are now standard features in financial NLP, and they came from this paper's insight that the relevant categories are financial, not emotional.

The honest limitations

• A dictionary still cannot read. Word counting has no notion of negation, context, sarcasm or comparison. "We do not expect losses" and "we expect losses" contain the same words. The paper improves the vocabulary, not the comprehension. Modern language models address this, at the cost of being far harder to audit.
• The word lists are a snapshot of a vocabulary. Corporate language evolves. New jargon appears. A list hand-built from filings of the 1990s and 2000s slowly drifts out of date, and it requires ongoing maintenance (which the authors have provided).
• The lists are hand-made judgements. Deciding whether a word is "negative in a financial context" is a human call. It is a well-informed, carefully-documented, transparent human call, which is far better than an unexamined borrowed one, but it is not objective truth.
• It is US-centric and filing-centric. The dictionaries were built from US 10-K filings in English. Applying them to earnings call transcripts, to social media, to non-US filings or to other languages carries exactly the same domain-mismatch risk that the paper was written to warn about.
• Signal does not mean profit. The paper shows the tone measures relate to returns, volatility and fraud. It is not a demonstration of a tradeable strategy net of costs, and the most obvious versions of that trade have long since been arbitraged.

The one-line takeaway

Loughran and McDonald showed that the psychology dictionary everyone was using to measure the tone of company filings misclassifies about three quarters of its "negative" words, because in finance a liability is not a tragedy and a cost is not a complaint, and by building a proper finance-specific dictionary they gave the entire field of financial text analysis a foundation it could actually stand on.