Maximum Entropy Distributions

You're told a stock's daily return has mean 0 percent and standard deviation 1.5 percent, and nothing else. Infinitely many distributions share exactly that mean and variance — a spiky one, a lopsided one, one with two separate humps. Which do you pick? Picking any of them beyond the two facts you were actually given means smuggling in an assumption you have no evidence for. Maximum entropy is the rule for refusing to smuggle anything in.

The analogy: don't claim to know more than you do

Entropy measures how "spread out" or unpredictable a distribution is — a spike at one value has zero entropy (fully predictable), a wide, flat shape has high entropy (maximally unpredictable). If you know only a few summary facts about a random quantity, choosing a distribution that is more concentrated than those facts force is choosing to believe extra things you weren't told. The maximum entropy distribution is the one with the highest entropy — the least concentrated, most honestly noncommittal shape — among all distributions consistent with the facts you actually have.

Writing it down

Let $H$ denote entropy, defined for a density $f$ as

$$H(f) = -\int f(x) \log f(x)\, dx .$$

In words: entropy weights each possible outcome by how surprising it is (the $-\log f(x)$ part) and averages that surprise over the distribution itself. Given a set of constraints — "the mean is $\mu$", "the variance is $\sigma^2$", "$x$ is always positive" — the maximum entropy distribution is the $f$ that makes $H(f)$ as large as possible while still satisfying every constraint exactly.

The remarkable part is that this abstract optimization has closed-form, familiar answers:

what you know	maximum entropy answer
support is $[a,b]$, nothing else	uniform on $[a,b]$
support is $(0,\infty)$, mean is $\mu$	exponential with rate $1/\mu$
support is $(-\infty,\infty)$, mean $\mu$, variance $\sigma^2$	normal $\mathcal{N}(\mu,\sigma^2)$
support is $\{0,1,2,\dots\}$, mean is $\lambda$	Poisson with rate $\lambda$

Read the third row again: it is the actual justification for using the normal distribution whenever you only trust a mean and a variance. It is not that returns are "naturally" bell-shaped — it's that the bell shape is the most agnostic choice available once those two numbers are fixed, and any other shape sneaks in unjustified structure.

Drag the mean and standard deviation above. Every curve you can produce this way has the same property: among all shapes sharing that particular mean and variance, this bell curve is the one adding the least extra information beyond those two numbers.

Worked example 1: mean known, nothing else

You're told a wait time between trade fills is always positive and averages 2 seconds. Solving the constrained optimization (Lagrange multipliers on $\int f = 1$ and $\int x f(x)\,dx = 2$) gives $f(x) = \lambda e^{-\lambda x}$ with $\lambda = 1/2 = 0.5$. Check it satisfies the constraint: $\int_0^\infty x \cdot 0.5 e^{-0.5x}\,dx = 1/0.5 = 2$. Correct. The exponential distribution isn't chosen because fill times are "naturally memoryless" — it's the least-assuming shape consistent with "positive, averages 2 seconds."

Worked example 2: mean and variance both known

A risk model is told a factor return has mean 0 and variance 4 (standard deviation 2), no other information. Maximizing entropy subject to both constraints, using Lagrange multipliers on $\int f=1$, $\int xf = 0$, $\int x^2 f = 4$, yields $f(x) \propto e^{-\lambda_2 x^2}$ — a Gaussian shape by construction, and solving for the multiplier that matches variance 4 gives exactly $\mathcal{N}(0, 4)$. Plug in $x=2$ (one standard deviation out): $f(2) = \frac{1}{\sqrt{2\pi \cdot 4}} e^{-2^2/(2\cdot 4)} = \frac{1}{5.013} e^{-0.5} \approx 0.121$. That single density value is the maximum-entropy model's best honest guess at how likely a return near +2 is, given only the two moments it was handed.

What this means in practice

• Risk models default to normal for exactly this reason: given only a mean and variance estimate, assuming normality is the most defensible choice, not the most realistic one — real returns have fatter tails, which means more information (skew, kurtosis, or the full historical distribution) is available and should be used instead of defaulting blindly.
• Bayesian priors are often built by maximum entropy: pick the least informative prior consistent with known constraints (support, mean), so the data — not the prior — does the work of updating your beliefs.
• Model risk shows up when someone assumes a distribution shape more specific than their actual information justifies, then reports false confidence from that extra, unearned structure.