Text Preprocessing and Tokenization

Before any model — a simple bag-of-words classifier or a modern large language model — can process text, the text has to be broken into discrete pieces and converted to numbers. That process is tokenization, and it's usually preceded by a round of cleanup called preprocessing, and both quietly determine what a downstream NLP model can and can't pick up on.

Preprocessing typically includes lowercasing text, stripping punctuation and boilerplate (page headers, disclaimers repeated in every filing), normalizing whitespace, and sometimes removing very common "stopwords" like "the" or "and" that carry little signal for older bag-of-words style models. Tokenization then splits the cleaned text into units — historically whole words, but modern models mostly use subword tokenization: common words stay whole ("trading" is one token) while rare or unfamiliar words get split into smaller recognizable pieces ("hyperinflationary" might become "hyper" + "inflation" + "ary"). This lets a model handle words it never saw during training without needing an entry for every possible word in its vocabulary.

The order and choices here matter more than they look. Aggressively stripping punctuation can delete meaningful signal — "we do not expect a decline" versus "we do not, expect a decline" turn into very different sentences if a stray comma changes the parse, and financial text is full of hedged, negation-heavy language where small words like "not" or "no longer" flip the entire meaning of a sentence. Lowercasing can also collapse real distinctions, like a company ticker that happens to spell an English word.

What this means in practice

Every NLP pipeline in finance — parsing central bank statements for hawkish/dovish tone, scanning SEC filings for risk-factor language, running sentiment on earnings-call transcripts — starts with this step, and a sloppy tokenization choice quietly caps how good any model built on top of it can be, no matter how sophisticated the model itself is. It's also the first thing to check when a fine-tuned model behaves oddly on financial text: many general-purpose tokenizers were built on everyday web text and handle ticker symbols, dollar figures, and financial jargon worse than they handle ordinary prose.