Elman and Jordan Recurrent Cells

A plain feedforward neural network has no memory: feed it the same input twice and it gives the same output twice, with no notion of what came before. For sequences — text, price series, sensor readings — that's a problem, since what happened a moment ago usually matters. Elman networks (1990) and Jordan networks (1986), the earliest simple recurrent architectures, fixed this by feeding a copy of the network's own recent activity back in as an extra input on the next time step, giving the network a crude form of memory.

The two designs differ in exactly what gets copied back. An Elman network feeds back the hidden layer's activations from the previous step — the network's internal state feeds itself. A Jordan network instead feeds back the previous output layer's activations — what the network actually predicted last time becomes part of this time's input. Elman's version is more general-purpose since the hidden layer can encode information that never surfaces in the output; Jordan's version is more natural when the previous prediction itself is genuinely informative about the next one, such as in some sequence-generation tasks.

Both architectures suffer from the same practical limitation: because the feedback loop is trained via backpropagation through time, gradients from many steps back tend to vanish or explode, so neither can learn dependencies spanning more than a handful of time steps reliably. This weakness is exactly what motivated LSTM and GRU cells, which add gating mechanisms to control what memory is kept or discarded.