Sentence Transformers and Contrastive Fine-Tuning

A raw BERT-style model, run on a single sentence, was never actually trained to produce a good summary vector for that whole sentence — it was trained to predict masked-out individual words using surrounding context. Averaging its internal word vectors together produces something usable but mediocre: two sentences that mean almost the same thing can end up further apart, by the model's own similarity measure, than two sentences that mean completely different things. Using an off-the-shelf language model's raw outputs as a similarity feature, without adapting it first, quietly bakes this mismatch into every downstream signal.

Sentence transformers fix this by fine-tuning a language model with an objective that directly targets what a finance desk actually wants: sentences known to be similar should end up with embeddings that are close together, and sentences known to be dissimilar should end up far apart.

How contrastive fine-tuning works

The model is trained on pairs (or triplets) of sentences with a known similarity relationship, using a loss function that directly rewards the geometry a downstream user needs:

• Positive pairs — two sentences that mean the same thing (paraphrases, or a sentence and its back-translation) — are pushed together in embedding space.
• Negative pairs — unrelated sentences, or in a triplet setup an "anchor," a "similar," and a deliberately dissimilar "hard negative" — are pushed apart.
• The loss (commonly a contrastive or triplet loss, see Cosine Embedding and Triplet Losses) is computed directly on cosine similarity between embeddings, so training explicitly optimizes for the metric that will later be used at inference time.

For finance specifically, the positive/negative pairs are often domain-specific: two earnings-call excerpts describing the same type of guidance change count as a positive pair, while an earnings excerpt and an unrelated macro commentary excerpt count as a hard negative — teaching the model to distinguish finance-relevant nuance a generic paraphrase dataset wouldn't cover.

Worked example

A team fine-tunes a base encoder using 50,000 pairs of financial-news sentence paraphrases as positives and randomly sampled unrelated sentences as negatives. Before fine-tuning, cosine similarity between "the firm lowered its earnings outlook" and "management cut full-year guidance" is $0.41$ — barely above the similarity of two random unrelated sentences at around $0.30$, because the base model wasn't trained to equate paraphrases. After ten epochs of contrastive fine-tuning, the same pair's similarity rises to $0.88$, while similarity to an unrelated control sentence ("the company opened a new regional office") drops to $0.09$. On a held-out validation set of 2,000 labeled pairs, the fraction correctly ranked (true paraphrase scored above the hardest unrelated distractor) rises from 61% before fine-tuning to 93% after — a large, measurable improvement in exactly the property document-embedding features depend on.

What this means in practice

Off-the-shelf embeddings from an un-adapted language model are a reasonable starting point, but a desk building similarity-based features (see Document Embeddings as Alpha Features) gets a large, measurable accuracy jump from contrastive fine-tuning on domain-specific pairs, often achievable cheaply with parameter-efficient methods like LoRA (see Parameter-Efficient Fine-Tuning and LoRA) rather than full retraining. The output is only as good as the pairs it was trained on, though — a model fine-tuned on generic paraphrase data will still underperform on finance-specific nuance it never saw examples of.