Schema Evolution and Versioning

A vendor feed adds a new field, "settlement currency," to their trade record format. Six months later, someone realizes the old field "price" was ambiguous about currency all along, and starts wanting to rename it. Meanwhile there are three years of historical files written in the old format sitting on disk, dozens of backtests that read that data expecting the old field names, and a live pipeline that can't simply stop while everyone agrees on a new format. Schema evolution is the practice of managing exactly this situation: how a data format changes over time without breaking the code and files that already depend on the old version.

The core distinction is between changes that are backwards-compatible and changes that aren't. Adding a new optional field is usually safe — old readers that don't know about it just ignore it, and new readers can handle records that lack it by falling back to a default. Removing a field, renaming a field, or changing a field's type (an integer becoming a float, a single value becoming a list) is not safe by default, because code written against the old schema will either crash or silently misinterpret the new data. A schema-evolution policy is really a set of rules about which kinds of changes are allowed casually and which require a deliberate migration.

Versioning is how a system tracks which schema a given piece of data was written under, so a reader knows how to interpret it. This can be as simple as a version number embedded in a file's metadata or a message's header, checked before any field is read. Without an explicit version, a reader has to guess the schema from the data's shape — which sometimes works and sometimes produces a confident, wrong interpretation, such as reading an old three-field record as if it were the new four-field format and silently shifting every value one column over.

Worked example: an additive change versus a breaking one

A trade record originally has fields (symbol, price, size). Version 2 adds venue as a new field with a default of "unknown" for old records — this is additive and backwards-compatible: any reader written for version 1 still works, ignoring the extra field, and any reader written for version 2 handles version-1 files correctly by filling in the default. Now suppose version 3 splits price into bid and ask. A version-2 reader that still expects a single price field will either crash on a version-3 record or, worse, silently read bid as if it were price, producing a plausible-looking but wrong number with no error at all. That second kind of change needs an explicit migration step and a version check, not a hopeful default.

What this means in practice

Prefer additive changes (new optional fields) over renaming or restructuring existing ones whenever possible, since additive changes don't require touching historical data or old readers. When a breaking change is genuinely necessary, write a migration that transforms old files into the new schema, keep an explicit version tag on every record or file, and make readers check that tag rather than assume. Skipping the version tag is the single most common way schema changes turn into silent data corruption instead of a loud, catchable error.