Structuring a Research Project Directory

A research project that starts as a single messy notebook tends to stay that way, and six months later nobody — including its author — can tell which cell produced which chart, or whether the reported result used the current version of the data. A standard directory structure fixes this by giving every kind of artefact a fixed home: raw data stays untouched in its own folder, code that transforms it lives separately, configuration (parameters, date ranges, universe definitions) is kept out of the code itself, and generated outputs (figures, tables, fitted models) go in a folder that could be deleted and regenerated at any time without losing anything irreplaceable.

The key discipline this enforces is a one-way flow: raw data is never edited in place, code reads from data and configuration and writes to outputs, and nothing is manually edited downstream of where the pipeline produces it. That way, rerunning the pipeline from scratch always reproduces the same outputs, and a reviewer can trace any number in a research report back to the exact code and data version that generated it.

Concrete illustration. A typical layout: data/raw/ (untouched source files), data/processed/ (cleaned, feature-engineered versions, regenerable from raw), config/ (YAML files specifying date ranges and universe filters), src/ (the actual pipeline code), notebooks/ (exploratory analysis only, not the source of truth for results), and outputs/ (figures and tables, timestamped, treated as disposable). A new team member can look at this layout and know exactly where to find the definition of "the current signal" without asking anyone.