Expected Calibration Error

A model predicts a 30% chance of a stock beating earnings estimates. Is that 30% meaningful, in the sense that stocks it labels "30%" actually beat estimates about 30% of the time — or is it just a score the model happens to output that correlates loosely with the right answer without meaning anything as a probability? A reliability diagram answers this visually, bucket by bucket. Expected Calibration Error (ECE) compresses that same picture into a single number, so two models — or the same model checked before and after a fix — can be compared by calibration quality without eyeballing a chart every time.

How ECE is built

Take the model's predicted probabilities on a validation set, sort them into $M$ equal-width bins (commonly 10, so bins of width 0.1), and in each bin compare the average predicted probability to the actual fraction of positives observed in that bin:

$$\text{ECE} = \sum_{m=1}^{M} \frac{n_m}{n} \left| \text{acc}(m) - \text{conf}(m) \right|$$

In plain English: for each bin, take the gap between what the model claimed (average confidence) and what actually happened (observed accuracy), weight that gap by how many predictions fell in the bin, and add up the weighted gaps across all bins. A perfectly calibrated model has ECE of 0 — every bin's average prediction matches its observed hit rate exactly. A model that's systematically overconfident or underconfident racks up a larger ECE, even if its predictions rank candidates correctly and would score well on AUC.

Worked example: computing ECE by hand on three bins

A model's predictions on 300 validation cases fall into three bins. Bin "0.1–0.4" (average predicted probability 0.25) has 100 cases, of which 15 turned out positive (observed rate 0.15) — a gap of $|0.25 - 0.15| = 0.10$. Bin "0.4–0.7" (average prediction 0.55) has 120 cases, 66 positive (observed rate 0.55) — a gap of 0. Bin "0.7–1.0" (average prediction 0.85) has 80 cases, 56 positive (observed rate 0.70) — a gap of $|0.85 - 0.70| = 0.15$.

$$\text{ECE} = \frac{100}{300}(0.10) + \frac{120}{300}(0) + \frac{80}{300}(0.15) = 0.033 + 0 + 0.040 = 0.073$$

An ECE of about 0.07 means the model's stated confidence is, on average, off by roughly 7 percentage points — well-calibrated in the middle bin, but overconfident at both the low and high ends, which a single AUC or accuracy number would never reveal since ranking can stay intact even as calibration drifts.

What this means in practice

ECE matters wherever a model's raw probability is used directly rather than just its rank — sizing a bet in proportion to predicted win probability, setting a reserve based on predicted default probability, or combining several models' outputs by averaging their probabilities. A model with a good ECE can be trusted to size decisions off its stated confidence; a model with poor ECE needs a calibration fix (like Platt scaling or isotonic regression) layered on top before its raw output is used for anything but ranking.