The No Free Lunch Theorem

It's tempting to ask "which model is best — gradient boosting, neural nets, linear regression?" as if there's a single true ranking. The No Free Lunch (NFL) theorem says that question has no universal answer: if you average an algorithm's performance across every possible dataset it could ever see, including all the weird, adversarial, randomly-labeled ones, every algorithm ties. A model that guesses randomly does exactly as well, on average over all possible worlds, as one carefully fit with cross-validation.

This sounds like it undermines machine learning, but the resolution is simple: real problems are never drawn uniformly from "every possible dataset." Stock returns, images, and language all have structure — smoothness, locality, redundancy — and a model that's a good match for that structure will consistently beat one that isn't, on the actual problems anyone cares about. NFL just formalizes that "best algorithm" is meaningless without specifying which family of problems you mean, which is why a factor model tuned on equities can be worthless applied blindly to crypto order-book data with completely different structure.

The practical takeaway is that model selection is really an assumption-selection exercise: choosing linear regression over a random forest is a bet about how smooth the true relationship is, not a claim that linear regression is objectively superior.