Neural Ordinary Differential Equations

A residual network updates a value layer by layer: $h_{t+1} = h_t + f(h_t, \theta)$, each layer nudging the previous state a little. What happens if you shrink the size of each nudge and add proportionally more layers to compensate, pushing the step size toward zero? The discrete staircase of layers starts to look like a smooth, continuous path — and that continuous limit is exactly what a differential equation describes.

The analogy: from footsteps to a flowing river

A ResNet's layer-by-layer updates are like a series of discrete footsteps across a landscape, each one a fixed-size hop. A Neural ODE instead describes the rate of change at every point — like describing a river's current rather than a series of hops — and leaves it to a numerical solver to figure out how finely to step through that flow to get from a starting point to an ending point.

The mechanics

Where a ResNet layer computes $h_{t+1} = h_t + f(h_t, \theta)$ (one discrete Euler step), a Neural ODE instead defines the derivative:

$$\frac{dh(t)}{dt} = f(h(t), t, \theta)$$

In words: instead of specifying what the state becomes after one fixed step, you specify how fast and in what direction it is currently changing, and an ODE solver integrates that rate from a start time $t_0$ to an end time $t_1$ to produce the output $h(t_1)$. "Depth" is no longer a fixed integer number of layers — it's continuous time, and the solver adaptively decides internally how many small steps it needs.

Worked example 1: integrating a toy ODE by hand

Take $\frac{dz}{dt} = -z$, $z(0) = 1$, integrated from $t=0$ to $t=1$ using simple Euler steps of size $0.5$: step 1, $z(0.5) = z(0) + 0.5 \times (-z(0)) = 1 - 0.5 = 0.5$. Step 2, $z(1) = z(0.5) + 0.5 \times (-z(0.5)) = 0.5 - 0.25 = 0.25$. The true analytic solution is $z(1) = e^{-1} = 0.368$ — the two-step Euler approximation ($0.25$) is close but not exact; a finer step size (more, smaller steps) would converge toward $0.368$, which is exactly the "adaptive" part a real ODE solver automates.

Worked example 2: a 3-layer ResNet as a coarse solver

A 3-layer ResNet computing $h_1 = h_0 + f(h_0)$, $h_2=h_1+f(h_1)$, $h_3=h_2+f(h_2)$ with the same $f$ at every layer is, by construction, exactly a 3-step Euler integration of $dh/dt=f(h)$ with step size $1$. A Neural ODE with the same $f$ integrated more finely (say, 20 adaptive steps) approximates the same underlying continuous trajectory more accurately — the ResNet is a specific, coarse, fixed-step-count special case of the continuous model.

What this means in practice

Because the output is defined by integrating to any target time, Neural ODEs naturally handle irregularly-sampled time series — like tick data with uneven gaps — by integrating directly to each observation's actual timestamp instead of forcing evenly-spaced discrete steps. Training also typically uses the adjoint method, which integrates the gradient backward through time without storing every intermediate solver step, saving memory versus backpropagating through a very deep unrolled network.