Information Criteria: AIC and BIC

Adding another predictor to a regression almost always improves the in-sample fit, even if that predictor is pure noise with no real relationship to the outcome — with enough variables, a model can fit historical data almost perfectly while capturing nothing generalizable. R-squared alone can't warn you about this because it mechanically rises (or at worst stays flat) every time you add a variable, no matter how useless. Information criteria fix this by scoring a model on both how well it fits and how many parameters it used to get there, penalizing complexity directly so that a needless extra variable can actually make the score worse, not better.

An analogy: a résumé that's padded versus one that's earned

Comparing two job candidates by raw résumé length is a bad idea — padding a résumé with filler bullet points always makes it longer, whether or not the extra content reflects real added value. A better evaluator asks "how much genuine substance is here, relative to the length," penalizing bulk that doesn't earn its place. Information criteria do exactly this to a statistical model: they don't just reward how well the model explains the data (like a résumé's raw content), they subtract a penalty for every extra parameter used (like docking points for filler), so a model only "wins" by adding a variable if the improvement in fit is worth more than the cost of the added complexity.

The mechanics, one symbol at a time

Let $L$ be the model's maximized likelihood (roughly, how probable the observed data is under the fitted model — bigger is a better fit) and $k$ be the number of estimated parameters. The Akaike Information Criterion is:

$$AIC = 2k - 2\ln(L),$$

and the Bayesian Information Criterion, for a sample of size $n$, is:

$$BIC = k\ln(n) - 2\ln(L).$$

In both formulas, lower is better — the $-2\ln(L)$ term rewards a better fit (higher likelihood shrinks this term), while $2k$ or $k\ln(n)$ penalizes each additional parameter. In plain English: both scores are a tug-of-war between "does adding this variable actually explain the data better" and "how much extra complexity did that cost," and you pick whichever candidate model has the lower score. The key difference is the size of the complexity penalty: BIC's penalty, $k\ln(n)$, grows with the sample size and for any reasonably large dataset ($n > 7$ or so) is stricter than AIC's flat $2k$ penalty — BIC tends to favor simpler models, especially as more data accumulates, while AIC is more willing to keep a variable if it helps the fit even a little.

Worked example 1: comparing two factor models

Two candidate models are fit to the same 100 monthly return observations. Model A uses 3 factors ($k=3$) and achieves $\ln(L) = -140$. Model B adds two more factors ($k=5$) and achieves a better-fitting $\ln(L) = -137$. Computing AIC for each:

$$AIC_A = 2(3) - 2(-140) = 6 + 280 = 286, \qquad AIC_B = 2(5) - 2(-137) = 10 + 274 = 284 .$$

Model B has the lower AIC (284 vs 286), so AIC prefers it — the two extra factors improved the fit enough to be worth their added complexity, by AIC's more lenient standard. Now check BIC, with $n=100$ so $\ln(100) \approx 4.605$:

$$BIC_A = 3(4.605) + 280 = 13.8 + 280 = 293.8, \qquad BIC_B = 5(4.605) + 274 = 23.0 + 274 = 297.0 .$$

BIC prefers Model A (293.8 vs 297.0) — the same two extra factors that AIC judged "worth it" fail BIC's stricter complexity penalty. This disagreement is normal and expected: AIC and BIC are answering subtly different questions (AIC aims to minimize prediction error; BIC aims to identify the true, most parsimonious model), and it's common for them to favor different-sized models on the same data.

Worked example 2: a variable that shouldn't have been added

A regression's log-likelihood improves only marginally, from $\ln(L) = -200$ to $\ln(L) = -199.6$, after adding one more parameter ($k$ goes from 4 to 5, $n=150$). The AIC change: old $AIC = 8 + 400 = 408$; new $AIC = 10 + 399.2 = 409.2$ — AIC goes up, meaning the tiny fit improvement (0.4 in log-likelihood) wasn't worth the added parameter's flat cost of 2. This is exactly the case information criteria are built to catch: R-squared would have ticked up slightly with the new variable and looked like an improvement, while AIC (and BIC even more emphatically) correctly flags that the added complexity isn't earning its keep.

Drag model complexity up and watch how in-sample fit keeps improving while a held-out measure eventually worsens — information criteria are a way of estimating that crossover point using only the training data's likelihood and parameter count, without needing a separate held-out sample.

What this means in practice

AIC and BIC are the standard, computationally cheap way to compare a small set of candidate models — how many lags to include in a time-series model, how many factors to keep in a regression, which of several nested specifications to report — without needing a separate held-out validation sample the way cross-validation does (see Cross-Validation for Time Series). They're most useful for comparing a handful of pre-specified candidates; for searching over many possible models or tuning a continuous penalty strength, methods like elastic net (see Elastic Net Regularization) or cross-validation are usually a better fit.