Data Engineering

Read these first

1. 01Tick Data vs Bar Data

   Tick data records every individual trade or quote update as it happens; bar data compresses a time window into open/high/low/close summary numbers. The choice between them is a bandwidth-versus-information tradeoff that shapes what a strategy can even see.

2. 02Columnar Storage and Parquet

   Columnar formats like Parquet store all values of one column together instead of one row at a time, which makes 'scan a billion timestamps and one price column' fast and compressible in a way row-oriented storage never can be.

3. 03Corporate Actions and Price Adjustment

   Splits, dividends and other corporate actions create artificial jumps in a raw price series. A price feed engineer applies backward-looking adjustment factors so that a return computed across the jump reflects what an investor actually earned, not a data artifact.

4. 04Point-in-Time Databases

   A point-in-time database can answer not just 'what is true now' but 'what did we believe was true as of last Tuesday,' which is the only honest way to backtest a strategy that would have used data as it looked in the past, restatements and all.

Then the rest

1. 05Event Time vs Ingest Time
2. 06Exchange Calendars and Trading Sessions
3. 07Level 1, Level 2 and Level 3 Data
4. 08Timestamp Conventions and Time Zones
5. 09Bad Tick and Outlier Detection
6. 10Bar Construction Methods
7. 11Batch vs Streaming Pipelines
8. 12Bitemporal Data Modeling
9. 13Data Quality Checks and Validation
10. 14ETL vs ELT Pipelines
11. 15Market Data Compression
12. 16Market Data Feed Handlers
13. 17Message Queues and Log-Based Streaming
14. 18Partitioning and File Layout
15. 19Schema Evolution and Versioning
16. 20Time-Series Databases
17. 21Universe Construction and Delistings