Annotation Guidelines and Inter-Annotator Agreement
Training data labelled by humans is only as good as the instructions those humans followed, and measuring how often independent annotators agree is the standard way to check whether the labels mean anything.
Before a sentiment or classification model can be trained, someone has to decide what "positive news for this stock" actually means for a few thousand example headlines — and if two people labelling the same headlines disagree constantly, the resulting training data is teaching the model noise rather than a signal. Annotation guidelines are the written rules that tell labellers exactly how to handle edge cases (a headline that's factually negative but already priced in, say), so that different people labelling the same text tend to produce the same label.
Inter-annotator agreement is the check on whether those guidelines actually worked: give the same batch of examples to two or more annotators independently, then measure how often they agree, typically with a statistic like Cohen's kappa that corrects for the agreement expected by pure chance.
Low agreement between independent annotators means the labelling task itself is ambiguous or the guidelines are unclear, and no amount of downstream modelling can fix training data whose labels don't reliably mean the same thing twice.
Worked example
Two annotators independently label 200 headlines as positive or negative for a stock. They agree on 170 of them — 85% raw agreement, which sounds solid. But if 80% of headlines in the sample are positive, two annotators guessing at random would already agree roughly of the time by chance alone, so the real signal in that 85% is much thinner than it first looks. Cohen's kappa adjusts for this by comparing observed agreement against the chance baseline, giving a more honest read on whether the guidelines are working.