BERT and Masked Language Models

The trouble with Word Embeddings and Word2Vec is that a word gets one vector forever. Charge in "a one-off charge to earnings" and charge in "the SEC filed a charge" collapse into a single averaged point, which is exactly wrong for anyone reading financial text, where the same word swings between accounting jargon and legal jargon sentence to sentence.

BERT's answer is to compute the representation of a word after looking at the sentence it sits in. Feed in "a one-off charge to earnings" and the vector for charge comes out near writedown. Feed in "the SEC filed a charge" and the same input word comes out near allegation. The representation is a function of the whole context, not a lookup.

The training trick that makes this possible is a cloze test — the fill-in-the-blank exercise from school. Take a sentence, hide about one word in seven, and make the model guess what was hidden using everything around it. Nobody labels anything; the corpus is its own answer key. Do that a few hundred million times over books, news and filings and the model is forced to learn grammar, world knowledge, and the fact that charge means different things in different neighbourhoods.

What "masked language modelling" actually does

Each input sentence is tokenised, and roughly $15\%$ of the tokens are selected for prediction. The model — a stack of transformer layers built on The Self-Attention Mechanism — produces a vector for every position, and at each selected position a small output layer scores every word in the vocabulary. Training pushes the score of the true word up and everything else down.

A second detail matters more than it looks. Of the selected tokens, only $80\%$ are actually replaced by a [MASK] placeholder. $10\%$ are swapped for a random word, and $10\%$ are left exactly as they were. The reason is that [MASK] never appears when you later use the model on real text, so a model that only ever computed good representations at mask positions would be useless in practice. Forcing it to sometimes correct a wrong word, and sometimes confirm a right one, keeps every position honest.

Worked example: scoring the blank

Suppose the sentence is "margins badly [MASK] the estimate" and the model's output layer produces these raw scores, or logits, for four candidate words: beat $4.0$, missed $2.0$, matched $1.0$, fell $0.5$.

Convert them to probabilities with a softmax — exponentiate each, then divide by the total:

$$e^{4.0} = 54.598, \quad e^{2.0} = 7.389, \quad e^{1.0} = 2.718, \quad e^{0.5} = 1.649$$

These sum to $66.354$, so

$$P(\text{beat}) = \frac{54.598}{66.354} = 0.823, \qquad P(\text{missed}) = \frac{7.389}{66.354} = 0.111 .$$

The remaining two get $0.041$ and $0.025$. In plain English: the model is fairly sure the blank is "beat". But the true word here is missed, because of the adverb badly. The training loss is the negative log of the probability assigned to the truth:

$$-\ln(0.111) = 2.198$$

A confident wrong answer is expensive, and the gradient from that loss is what teaches the model that badly flips the polarity of what follows. Repeat across a corpus and the model learns the pattern without anyone writing a rule.

Worked example: how much signal per pass

Take a training passage of $512$ tokens. Selecting $15\%$ gives

$$0.15 \times 512 = 76.8 \approx 77 \text{ tokens}$$

to predict. Of those, $80\%$ become [MASK] — about $61$ tokens — while roughly $8$ are replaced by a random word and $8$ are left untouched.

So a single pass over 512 tokens produces only 77 learning signals. A left-to-right model predicting every next token gets 512 from the same passage, nearly seven times more. That arithmetic is the honest cost of bidirectionality: BERT-style models need substantially more compute per unit of text, and it is why masked models are usually smaller than their generative cousins for a given training budget.

Using it on financial text

Pre-training is done once, expensively, by someone else. What you do is fine-tuning: bolt a small classifier onto the special [CLS] position that summarises the whole passage, then train on your own labelled data — a few thousand annotated sentences from earnings calls is often enough, versus the hundreds of thousands a from-scratch model would need. Typical jobs are sentiment on a per-sentence basis, risk-factor classification in 10-K filings, and pulling structured events out of press releases.

A general-purpose BERT has read very little accounting language, so a domain-adapted variant like FinBERT — further pre-trained on filings and calls before fine-tuning — usually beats it. See FinBERT and Domain-Adapted Models.