Hyperparameters vs Parameters

A trading desk has two kinds of numbers on its books. There are position sizes and hedge ratios recalculated every day from fresh data — those adjust automatically as conditions change. And there are the rules of the desk itself — max leverage, the stop-loss threshold, how many days of history the risk model looks back over — decided in advance, which daily recalculation never touches. Get those rules wrong and no amount of correct daily recalculation saves you. A machine learning model has exactly this split, and the two kinds of numbers have different names.

Parameters are the numbers a model fits from the training data. In linear regression these are the coefficients $\beta$; in a neural network they are every weight and bias; in a decision tree they are the specific split points chosen at each node. Nobody sets these directly — an optimization procedure like gradient descent searches for the values that minimize the training loss, and the model literally cannot be trained without discovering them.

Hyperparameters are the numbers a human (or an automated search) fixes before training begins and that training itself never touches. The learning rate in gradient descent, the number of trees in a random forest, the maximum depth of a single tree, the regularization strength $\lambda$, the number of layers in a network — all of these are chosen ahead of time and stay fixed while the parameters are being learned. Change a hyperparameter and you are effectively training a different model from scratch, not continuing the same one.

The mechanical test: does the loss function's gradient touch it?

The cleanest way to tell them apart: parameters appear inside the loss function being differentiated during training — gradient descent adjusts them directly because the loss is a function of them. Hyperparameters shape the loss function or the optimization procedure itself but never appear as a variable being solved for.

$$\hat{\theta} = \arg\min_{\theta} \; \Big[ \underbrace{L(\theta; X, y)}_{\text{fit to data}} + \underbrace{\lambda \, R(\theta)}_{\text{penalty}} \Big]$$

In words: training finds the parameter values $\theta$ that minimize a loss $L$ (how badly the model fits the data) plus a penalty term $R(\theta)$ scaled by $\lambda$. The search happens over $\theta$ — that's what "$\arg\min_\theta$" means, find the $\theta$ that minimizes this. But $\lambda$ itself is never searched over by this equation; it sits outside the minimization, fixed as a constant chosen beforehand. $\theta$ is a parameter. $\lambda$ is a hyperparameter.

Drag the learning rate here and watch the descent path change completely — crawling, converging cleanly, or overshooting into oscillation. The learning rate is a textbook hyperparameter: it controls how the parameters get found, but it is never itself among the numbers being solved for by the descent.

Worked example 1: ridge regression, counted out

Fit ridge regression, $y = \beta_0 + \beta_1 x_1 + \beta_2 x_2$, with penalty strength $\lambda = 0.5$, on a small dataset. Training runs and returns $\beta_0 = 1.2$, $\beta_1 = 0.8$, $\beta_2 = -0.3$. Those three numbers are parameters — they came out of the optimization, different data would produce different values, and re-running training on the same data with the same $\lambda$ reproduces them. $\lambda = 0.5$ is a hyperparameter — it was chosen before fitting (perhaps by cross-validation over a grid like $\{0.1, 0.5, 1, 5\}$), and the fitting procedure never adjusts it; if you want a different $\lambda$, you rerun training entirely, you don't "continue" the old fit.

Worked example 2: counting a small neural network

A network has one hidden layer of 8 units, ReLU activation, trained with Adam at learning rate $0.001$ for 50 epochs, dropout $0.2$. Count the parameters: with 5 input features, the hidden layer has $5\times8=40$ weights plus 8 biases $=48$; the output layer has $8$ weights plus 1 bias $=9$. Total: $57$ parameters, every one a number training searches over. Now count the hyperparameters fixed beforehand: layer count, units per layer, activation choice, optimizer, learning rate, epoch count, dropout rate — seven human decisions, none of which the 57-parameter optimization ever revisits.

What this means in practice

Hyperparameters have to be chosen using data the model's final performance will be judged on being kept separate from — typically a validation set distinct from both the training set (where parameters are fit) and the test set (where the finished model is judged once, at the end). Skipping this and picking the learning rate, tree depth, or lookback window by watching how well it does on the same data used for both fitting and final evaluation quietly turns the hyperparameter search into another form of overfitting, just one layer up from the parameters themselves.