Estimating Specific (Idiosyncratic) Risk

A factor model explains a stock's risk as exposure to market, size, value, and so on. But no factor model, however rich, ever explains 100% of a stock's variance. A pharmaceutical company can plunge 30% overnight on a failed drug trial that had nothing to do with the market, size, or value that day. That leftover, unpredictable, stock-specific piece of risk is called specific risk (or idiosyncratic risk), and a portfolio manager who ignores it is missing a large chunk of the actual risk in a concentrated book.

The analogy

A weather forecaster predicts a city's temperature using season, latitude, and elevation, general factors that explain most of the variation across cities. But even with a perfect model of those factors, one specific day in one specific city can still be freakishly hot or cold due to a local, unpredictable weather system. That local surprise is not a flaw in the forecaster's model, it is an irreducible piece of variability that exists for every city, every day, and any honest risk assessment has to add it back in rather than pretend the factor model alone tells the whole story.

Building the estimate

After fitting a factor model $R_i = \sum_k \beta_{i,k} f_k + \epsilon_i$, the residual $\epsilon_i$ is the part of stock $i$'s return the factors could not explain. Specific risk is simply the standard deviation of that residual, estimated from a rolling history:

$\sigma_{\epsilon,i} = \sqrt{\frac{1}{T-1}\sum_{t=1}^{T}(\epsilon_{i,t} - \bar{\epsilon}_i)^2}.$

In words: run the factor regression over many periods, collect the leftover residual each period, and measure how much those residuals typically bounce around. A stock with $\sigma_{\epsilon,i} = 3\%$ monthly has 3% of unexplained noise every month on top of whatever the factors predict; a stock with $\sigma_{\epsilon,i} = 0.5\%$ is almost entirely explained by the factor model. Total portfolio variance then combines both pieces:

$\sigma_p^2 = w^\top B \Omega B^\top w + w^\top D w,$

where $B$ is the exposure matrix, $\Omega$ is the factor covariance matrix, and $D$ is a diagonal matrix of each stock's specific variance $\sigma_{\epsilon,i}^2$. In plain words: total risk is factor risk plus specific risk, added because the model assumes specific risks are uncorrelated across stocks, each company's idiosyncratic surprise is its own separate event.

Picture the residual $\epsilon_i$ for one stock as a draw from a curve like this every period, centered at zero if the factor model is unbiased, with its spread, the standard deviation, being exactly the number specific-risk estimation is trying to pin down.

Worked example 1: two stocks, same factor exposure, different specific risk

Two stocks have identical factor exposures and therefore identical factor risk of $12\%$ annualized. Stock A has specific risk $\sigma_{\epsilon,A} = 5\%$, stock B (a small, thinly-traded biotech) has $\sigma_{\epsilon,B} = 25\%$. Total risk, combining independently (variances add): $\sigma_A = \sqrt{12^2+5^2} = \sqrt{144+25} = \sqrt{169} = 13\%$, while $\sigma_B = \sqrt{144+625} = \sqrt{769} \approx 27.7\%$. Despite sharing the exact same factor profile, stock B is more than twice as risky in total, entirely due to specific risk.

Worked example 2: diversification's effect on a portfolio

Hold 50 stocks, each with specific risk $20\%$, equal weight $1/50$, specific risks uncorrelated across names. Portfolio specific variance is $50 \times (1/50)^2 \times 0.20^2 = 0.0008$, so specific risk is $\sqrt{0.0008} \approx 2.8\%$. Holding just 5 of those same stocks equally weighted gives $5 \times (1/5)^2 \times 0.04 = 0.008$, so $\sqrt{0.008} \approx 8.9\%$. Going from 5 to 50 names cuts portfolio-level specific risk by more than two-thirds, a diversification benefit factor risk never gives you.

What this means in practice

Concentrated portfolios and single-stock positions are dominated by specific risk, not factor risk, which is exactly why a risk report showing "your book has only 0.9 market beta, low risk" can be dangerously misleading for a 10-stock book, the specific-risk column is where the real danger often hides.