Logistic Regression

Ordinary regression predicts a number. But a huge share of real questions are yes/no: will this loan default? will the trade hit its profit target before its stop? did the click convert? For those, you don't want a raw number, you want a probability, something pinned between 0 and 1. Logistic regression is the standard, dependable way to get one.

The obvious fix, fit a straight line and call the output a probability, breaks immediately: a line runs off to $+\infty$ and $-\infty$, so it happily predicts probabilities of $1.4$ or $-0.3$, which are nonsense. Logistic regression solves this by running the linear score through an S-shaped squashing curve (the logistic, or sigmoid, function) that gently flattens both ends so the output can never leave the $[0,1]$ range.

The model

You still build a linear score from your features, $z = \beta_0 + \beta_1 x_1 + \dots + \beta_k x_k$. Then you pass it through the logistic function to turn it into a probability:

$$p = \frac{1}{1 + e^{-z}}, \qquad z = \beta_0 + \beta_1 x_1 + \dots + \beta_k x_k.$$

Here $p$ is the predicted probability of a "yes", the $\beta$s are the coefficients you fit, and $x$s are your features. When the score $z$ is large and positive, $e^{-z}$ is tiny and $p$ approaches $1$; when $z$ is very negative, $p$ approaches $0$; and at $z = 0$ the probability is exactly $0.5$. The relationship becomes perfectly linear if you look at it in the right units, the log-odds:

$$\log\!\frac{p}{1-p} = \beta_0 + \beta_1 x_1 + \dots + \beta_k x_k.$$

That is the key to reading the model: the coefficients are linear effects on the log-odds, not on the probability directly.

How it's fit and how to read a coefficient

There is no tidy closed-form formula like Ordinary Least Squares (OLS) has. Instead you fit by Maximum Likelihood Estimation (MLE): choose the $\beta$s that make the observed yes/no outcomes most probable, found by a quick iterative solver. Once fit, each coefficient has a clean interpretation through the odds ratio. Increasing a feature by one unit multiplies the odds by $e^{\beta}$. A coefficient of $\beta = 0.7$ means each extra unit multiplies the odds of a "yes" by $e^{0.7} \approx 2$, it doubles the odds.

Worked example: probability of default

You model loan default with one feature, the borrower's debt-to-income ratio $x$ (as a decimal). The fit returns $\beta_0 = -4$ and $\beta_1 = 6$. What's the default probability for a borrower with a debt-to-income ratio of $0.5$?

First the score: $z = -4 + 6 \times 0.5 = -1$. Then the sigmoid:

$$p = \frac{1}{1 + e^{-(-1)}} = \frac{1}{1 + e^{1}} = \frac{1}{1 + 2.718} \approx 0.269.$$

So about a $27\%$ default probability. Bump the ratio to $0.7$: the score becomes $z = -4 + 6 \times 0.7 = 0.2$, and $p = 1/(1+e^{-0.2}) \approx 0.55$, now more likely than not to default. Notice the coefficient $\beta_1 = 6$ says every $0.1$ increase in debt-to-income adds $0.6$ to the log-odds, multiplying the odds by $e^{0.6} \approx 1.82$. Small changes near the middle of the curve swing the probability a lot; changes far out in the flat tails barely move it.

Where it misleads

• Perfect separation. If some feature splits the classes cleanly (all defaults above a threshold, none below), the MLE tries to push a coefficient to infinity and the fit blows up. Regularization (Ridge and LASSO Regularization) or a prior tames it.
• Probabilities need calibration. A model can rank cases well (good AUC) yet output probabilities that are systematically too high or too low. If you'll act on the number, not just the ranking, check calibration before trusting it.
• Still a linear model at heart. It's linear in the log-odds, so it can't capture interactions or curved effects unless you engineer those features in. When the boundary is genuinely complex, tree ensembles or gradient boosting usually beat it.
• Imbalanced classes. Rare events (defaults, fraud) make the intercept dominate and accuracy misleading, a model predicting "never" can be $99\%$ accurate and useless. Weight the classes or use the right metric.

Logistic regression is the honest first model for any classification problem: fast, interpretable, and hard to badly overfit. In quant workflows it's the natural engine for Meta-Labeling, deciding whether to act on a primary signal, and a clean baseline that fancier classifiers must beat to earn their complexity.