Identifiability of a Model

Before you ask "how do I estimate this parameter accurately," there's a more basic question: can it be estimated at all, even with infinite data? Some models are set up so that two genuinely different parameter values produce the exact same distribution of observable outcomes — no amount of data, however large, can ever tell them apart. That is a failure of identifiability, and it is a problem with the model, not with the sample size.

An analogy: two chefs, one dish, no receipt

Imagine a dish's saltiness depends only on the total amount of salt and soy sauce added together — the two ingredients are interchangeable for the outcome you can taste. You are handed the finished dish and asked: "how much salt, specifically, did the chef use?" No amount of tasting, no matter how careful, can answer that. Two completely different recipes — all salt, or all soy sauce, or any split between them — that sum to the same total produce an indistinguishable dish. The total is identifiable; the individual split is not. That is exactly the structure of a non-identifiable model: some combination of parameters is recoverable from the data, but the individual pieces are not.

The definition, one symbol at a time

Let $\theta$ ("theta") denote the model's parameters, and let $P_\theta$ denote the probability distribution of the observable data implied by a given value of $\theta$. The model is identifiable if

$$P_{\theta_1} = P_{\theta_2} \implies \theta_1 = \theta_2$$

Read this as: "if two parameter values produce the exact same distribution of what you can actually observe, then those two parameter values must have been the same value to begin with." In plain English: identifiability means different parameters always leave different fingerprints in the data. If it fails, there exist $\theta_1 \ne \theta_2$ with $P_{\theta_1} = P_{\theta_2}$ — two truly different worlds that look statistically identical from where you're standing, no matter how much data you collect from either one.

Why "more data" cannot fix it

This is the detail that trips people up. Ordinary estimation error — the wobble in your estimate $\hat\theta$ around the truth $\theta$ — shrinks as your sample size $n$ grows, roughly like $1/\sqrt{n}$, because random noise averages out. Non-identifiability is not noise. It is a structural tie between two parameter values baked into the model itself; the likelihood function is genuinely flat, or has two equal peaks, no matter how many observations feed into it. A bigger dataset makes each peak sharper and more certain — but it cannot tell you which peak, or which mixture of the two, is correct, because they were built to look the same.

The explorer above shows a running average settling onto the true value as $n$ grows — that settling only works because the parameter itself is identifiable from the data. For a non-identified parameter, no amount of extra data narrows the envelope around the truth; it stays exactly as wide as when you started.

Now picture the log-likelihood, not as a single-bottomed curve like the one above, but flattened along one direction into a trough with no unique minimum — every point along the flat floor of the trough fits the data equally well. A curve with one clean bottom, as pictured, always identifies a unique parameter; a curve with a flat floor never does, no matter how deep or well-sampled the trough is.

Worked example 1: a genuinely unidentified linear model

Suppose you model an asset's expected return as $E[Y] = \alpha + \beta_1 X_1 + \beta_2 X_2$, but your two predictors are perfectly collinear: $X_2 = 2 X_1$ always, in every observation you will ever see (say, both are the same signal reported in different units). Then $\beta_1 X_1 + \beta_2 X_2 = \beta_1 X_1 + \beta_2 (2X_1) = (\beta_1 + 2\beta_2) X_1$.

The data only ever reveals the combined quantity $\beta_1 + 2\beta_2$, never $\beta_1$ and $\beta_2$ separately. Concretely: if the true values are $\beta_1 = 3, \beta_2 = 1$, the combined coefficient is $3 + 2(1) = 5$. But $\beta_1 = 1, \beta_2 = 2$ gives the identical combined coefficient $1 + 2(2) = 5$. Both parameter pairs generate the exact same predictions for every possible value of $X_1$. Fit this model to a million observations, and $\beta_1$ and $\beta_2$ remain forever unrecoverable individually — only their fixed combination is identified.

Worked example 2: making it identifiable by adding a constraint

The mixture model $Y \sim p \cdot N(\mu_1, 1) + (1-p) \cdot N(\mu_2, 1)$ — a coin-flip mixture of two normal distributions — has a subtler problem: relabeling which component is "1" and which is "2" produces the same overall distribution. The pair $(p=0.3, \mu_1=5, \mu_2=10)$ generates the identical mixture as $(p=0.7, \mu_1=10, \mu_2=5)$ — just the labels swapped, with no way for the data to say which is "really" component 1.

The fix is a labeling constraint: require $\mu_1 < \mu_2$ by convention. Now $(p=0.3, \mu_1=5, \mu_2=10)$ is the only pair satisfying both the constraint and the observed mixture — the label-swap twin $(p=0.7, \mu_1=10, \mu_2=5)$ violates $\mu_1 < \mu_2$ and is ruled out by fiat, not by data. This is a common real pattern: identifiability is often restored not by collecting different data, but by adding a normalization or ordering rule that breaks a symmetry the model would otherwise have.

What this means in practice

• Multicollinear regressors in a factor model are a live version of Worked Example 1 — two highly correlated signals (say, two momentum lookback windows) leave their individual coefficients nearly unidentified even though their sum is well estimated; this is why regression output can show wildly unstable coefficients with tiny changes in the data even as the overall fit stays stable.
• Regime-switching and mixture models in finance (bull/bear regime detection, jump-diffusion parameter fits) routinely hit label-switching non-identifiability, exactly like Worked Example 2 — always check that the fitting procedure enforces an ordering or normalization.
• Before trusting any fitted parameter individually (not just the model's overall fit), check whether the likelihood is flat in some direction of parameter space — a flat direction means that parameter, or that combination of parameters, isn't pinned down by the data at all.