How Complicated Should Your Model Be? Akaike's Answer

Here is a fact that sits at the root of nearly every mistake in quantitative research: a more complicated model always fits the data you already have better than a simpler one.

Always. Not usually. Always. Add another lag to your ARIMA and the fit improves. Add another factor to your regression and the R-squared goes up. Add another parameter to anything and the errors get smaller. If you select models by how well they fit your historical sample, you will always select the most complicated model available, and you will always be wrong.

The reason is that some of that improved fit is real, and some of it is the model memorising the noise. And from inside the sample, those two look identical.

Hirotugu Akaike's 1974 paper gave the world a principled way to tell them apart, or at least to price the difference. His criterion, now called the AIC, appears in the output of essentially every statistical model ever fitted.

The problem: fit was measuring the wrong thing

Before Akaike, model selection in time-series work was largely a matter of sequential hypothesis testing and judgement. You would fit a model with two lags, then one with three, then test whether the extra lag was "significant." This works, sort of, but it has problems. The order in which you test matters. The significance level you use is arbitrary. And there is no coherent way to compare two models that are not nested inside each other.

Akaike's reframing was philosophical before it was mathematical. He pointed out that the goal is not to find the model that best describes the data you have. The goal is to find the model that will best describe data you have not seen yet. Those are completely different objectives, and confusing them is the original sin of model fitting.

Once you state the goal that way, the question becomes: given only the data I have, can I estimate how well each candidate model would perform on fresh data?

The key idea via analogy: the tailor who measures one customer

Imagine a tailor who fits a suit to a single customer by taking a thousand measurements: not just chest and inseam, but the exact position of a temporary crease in the shirt, the way the customer happened to be standing that afternoon, the slight swelling in one ankle from the walk over.

The suit will fit that customer, on that day, perfectly. Better than any suit ever made. But it will fit nobody else, and it will not even fit the same customer next Tuesday. The tailor has captured not just the customer's shape but the accidents of the moment.

A simpler tailor, using ten measurements, produces a suit that fits slightly less perfectly today but will still fit next Tuesday, and would roughly fit the customer's brother too.

Every extra parameter in your model is another measurement of a temporary crease. Some of them capture real structure. Some capture accidents. And in-sample fit cannot distinguish them.

Akaike's insight, which came from information theory, was that you can quantify how much fit improvement you should expect from pure accident. Adding a parameter to a model that has no real use for it still improves the in-sample fit by a predictable amount, on average. So if you know that amount, you can subtract it. Only the improvement above and beyond what noise would have given you for free counts as evidence.

The resulting formula is startlingly simple:

Score = (how badly the model fits) + (a penalty proportional to the number of parameters).

You compute this for every candidate model, and you pick the lowest score. That is it. A model earns its extra parameter only if the fit improves by more than the penalty. The genuinely deep part is that the size of the penalty is not arbitrary. Akaike derived it, from an argument about the expected information loss when you use an approximate model to represent reality. It is not a knob you tune. It is a number that falls out of the theory.

Why it mattered

• It made model selection mechanical and comparable. You can rank any set of candidate models, nested or not, on a single scale. There is no arbitrary significance level, no order-dependence, no argument.
• It completed the Box-Jenkins loop. The classic time-series workflow is: identify a candidate model, estimate it, check it. AIC gives you a principled way to do the identification step, choosing how many autoregressive and moving-average terms a series actually needs. It is the reason automated ARIMA-fitting routines exist.
• It is how you pick the lag length in a VAR. Every vector autoregression requires a decision about how far back to look. AIC (and its relatives) is the standard answer.
• It named and priced the overfitting tradeoff. Before Akaike, "do not overfit" was folk wisdom. After Akaike, there was an exchange rate: each parameter costs exactly this much, and must repay it in fit. That is a genuine intellectual advance, and it foreshadowed the entire modern machine-learning conversation about regularisation and the bias-variance tradeoff.
• It escaped statistics entirely. AIC is used in ecology, medicine, engineering, and machine learning. Akaike is one of the most cited authors in the history of science largely on the strength of this idea.

The honest limitations

AIC is a tool, not an oracle, and the ways it fails are worth knowing.

• It does not try to find the "true" model, and it will not. This is the subtlest point. AIC is designed to select the model with the best predictive performance, not the model that is actually correct. As your sample grows, AIC will keep selecting models that are slightly too big, because a spurious extra parameter costs little in prediction. If your goal is to identify the true structure, the BIC (Schwarz's criterion, which uses a heavier penalty that grows with sample size) is the more appropriate choice. AIC and BIC answer different questions, and treating them as competing answers to the same question is a persistent confusion.
• It only ranks the models you gave it. AIC will faithfully tell you which of your ten candidate models is best. If all ten are garbage, it will hand you the best piece of garbage with complete confidence and no warning. It is a relative measure, not an absolute one.
• Small samples break the penalty. The derivation is asymptotic. In small samples, the standard penalty is too light and AIC overfits badly. A corrected version (AICc) exists and should be used whenever the number of observations is not comfortably larger than the number of parameters. Skipping this correction is a common and consequential mistake.
• It assumes the data is what you say it is. AIC is built on the likelihood, so if your likelihood is wrong (you assumed normal errors and the data has fat tails, which describes essentially all financial data) then the AIC comparison inherits that error.
• It counts parameters, not effective complexity. A model with three carefully constrained parameters and a model with three free ones get the same penalty, even though they are not equally flexible. In modern machine learning, where "number of parameters" is a terrible proxy for complexity, this limitation is fatal and AIC is largely replaced by direct cross-validation.
• It does not save you from data mining across specifications. If you try five hundred model variants and pick the best AIC, you have overfitted your model selection, and no criterion computed on the same data can catch that.

The one-line takeaway

Akaike showed that the fit improvement you get from adding a parameter is partly real and partly a gift from noise, that the size of the gift can be calculated exactly, and that if you charge every parameter that price before comparing models, you get a defensible answer to the oldest question in modelling: how complicated should this be?