Entropy and Information

Entropy is a one-number answer to "how uncertain is this?" A coin you know will land heads has no uncertainty; a fair coin has some; a fair roulette wheel has a lot. Shannon's insight was that these can be put on a common scale — measured in bits — and that the very same number answers three questions at once: how surprised you are on average by the outcome, how much information you gain by learning it, and how many yes/no questions it takes to identify it. High entropy means unpredictable; zero entropy means you already knew the answer.

The formula

For an outcome with possible values of probability $p_1, p_2, \dots$, the (Shannon) entropy is

$$H = -\sum_i p_i \log_2 p_i.$$

Each $p_i$ is the probability of one outcome, and $\log_2$ (log base 2) is what makes the units come out in bits. The quantity $-\log_2 p_i$ is the surprise of outcome $i$: rare outcomes (small $p_i$) are very surprising, near-certain ones barely register. Entropy is just the expected surprise — the average, weighted by how often each outcome occurs. It bottoms out at 0 when one outcome is certain, and it is largest when every outcome is equally likely.

Worked example: coins and dice

A fair coin. Two outcomes, each probability $\tfrac12$:

$$H = -\big(\tfrac12 \log_2 \tfrac12 + \tfrac12 \log_2 \tfrac12\big) = -\big(\tfrac12(-1) + \tfrac12(-1)\big) = 1 \text{ bit}.$$

One bit — exactly one yes/no question ("heads?") settles it.

A biased coin, 90% heads. Now the surprises are lopsided:

$$H = -\big(0.9\log_2 0.9 + 0.1\log_2 0.1\big) = -\big(0.9(-0.152) + 0.1(-3.322)\big) = 0.137 + 0.332 = 0.469 \text{ bits}.$$

Under half a bit. A lopsided coin is easy to predict, so learning the result tells you less. A fair six-sided die, by contrast, has $H = \log_2 6 \approx 2.585$ bits — more outcomes, more uncertainty, so it takes between two and three yes/no questions on average to identify the face.

Where the intuition slips

• Values don't matter, only probabilities do. Entropy sees a fair coin that pays a penny and one that pays your life savings as identical — both are 1 bit. It measures unpredictability, not stakes or magnitude. Never confuse high entropy with high risk in dollar terms.
• Units are a convention. Using $\log_2$ gives bits; natural log gives "nats." The numbers scale by a constant, so always note the base before comparing.
• Continuous entropy is a different animal. Extending entropy to continuous variables (differential entropy) can go negative and depends on your units, so the clean "always non-negative, zero means certain" story is a discrete-only property.

Entropy is the foundation of information theory and the anchor for its most useful cousin, KL divergence, which measures how far one distribution sits from another. It also underlies information-theoretic signal and feature selection, and it connects to maximum likelihood through the idea of coding data as compactly as its uncertainty allows.