Getting Market Data for Free (and Its Traps)

Free market data is a genuine gift, you can build a serious research pipeline for $0. It is also a minefield, because free data is optimized for looking at charts, not for backtesting. Every trap below produces a backtest that runs cleanly, prints a plausible Sharpe, and is quietly wrong. The corruption is invisible unless you know exactly what to check. This guide shows how to pull and store data from the main free sources, then spends most of its length on the traps, because pulling data is easy and trusting it is where beginners get destroyed.

The free sources

Source	Covers	Auth	Watch out for
yfinance	Global equities, ETFs, indices, FX, some crypto	None	Unofficial Yahoo scrape; can break, rate-limit, or return silent gaps
FRED	Macro/rates/econ (CPI, GDP, yields)	Free key	Many series are restated, see point-in-time trap
Alpha Vantage	Equities, FX, crypto, fundamentals	Free key	25 requests/day, 5/min on free tier
Tiingo	EOD equities, crypto, news; clean adjustments	Free key	~500 symbols/hour free; excellent adjustment quality
ccxt	Crypto OHLCV from 100+ exchanges	Per-exchange	Exchange-specific quirks, 24/7 sessions, no split logic needed

A fuller rundown with links and current limits lives in the Tools & Data Directory. For US equities start with yfinance; graduate to Tiingo when adjustment quality matters; use FRED for macro and ccxt for crypto.

Pulling and storing efficiently

The pattern is always the same: pull once, store to disk, load from disk thereafter. Re-hitting the API on every run is slow, hammers rate limits, and, because free feeds change, makes your pipeline non-reproducible. Store in parquet, not CSV: it is columnar, compressed, ~5–10× smaller, an order of magnitude faster to read, and, crucially, it preserves dtypes and the datetime index instead of stringifying everything.

import yfinance as yf
from pathlib import Path

RAW = Path("data/raw"); RAW.mkdir(parents=True, exist_ok=True)

# auto_adjust=True returns split- AND dividend-adjusted OHLC (total-return prices)
df = yf.download("SPY AAPL", start="2015-01-01", auto_adjust=True)
df.to_parquet(RAW / "eq_prices.parquet")

Here is a self-contained round-trip you can run without a network, proving parquet preserves the index and dtypes exactly (the property CSV silently loses):

import numpy as np, pandas as pd
from pathlib import Path

dates = pd.bdate_range("2024-01-01", periods=6)
df = pd.DataFrame({"close": np.linspace(50, 55, 6), "volume": [1e6]*6}, index=dates)
df.index.name = "date"

df.to_parquet("prices.parquet")          # needs pyarrow (or fastparquet) installed
back = pd.read_parquet("prices.parquet")
print("round-trip identical:", df.equals(back), "| index kept:", back.index.name)
# -> round-trip identical: True | index kept: date

Trap 1, adjusted vs raw close, and why total return matters

This is the trap that ruins more beginner backtests than any other. When a stock does a 2:1 split, its raw price halves overnight, but nothing happened economically; you just own twice as many shares. If you compute returns off raw close, the split looks like a −50% crash:

import pandas as pd
dates = pd.bdate_range("2024-01-01", periods=6)
raw = pd.Series([100., 102., 101., 103., 52., 53.], index=dates)   # 2:1 split on day index 4

print("raw return on split day:", round(raw.pct_change().iloc[4], 4))   # -0.4951  (FAKE)

# back-adjust: scale every PRE-split price by the split factor (÷2)
adj = raw.copy(); adj.iloc[:4] *= 0.5
print("adjusted return on split day:", round(adj.pct_change().iloc[4], 4))  # +0.0097 (real)

The raw series manufactures a −49.5% return that never existed; the adjusted series shows the true +1%. Dividends work the same way: on the ex-dividend date the price drops by roughly the dividend, and only a total-return (dividend-adjusted) series captures the cash you actually received. Over decades, reinvested dividends are a large fraction of equity total return, ignoring them understates performance dramatically.

The practical rule: use the adjusted/total-return series for computing returns and signals (auto_adjust=True in yfinance). But know the catch, adjusted prices are retroactively rewritten every time a new split or dividend occurs, so yesterday's adjusted price is not the number you could have traded at. For anything touching order sizing, position value, or limit prices, keep the raw close too. Use adjusted for returns; use raw for "what price could I actually have transacted at." See Transaction Costs for why the traded price matters.

Trap 2, survivorship in free indices

Ask a free source for "the S&P 500" and you get today's 500 members. The companies that were in the index in 2005 and later went bankrupt, got delisted, or were acquired are simply gone. Backtest a strategy on today's constituents and you have implicitly selected for survivors, firms that didn't blow up, which flatters returns and hides risk. This is Survivorship Bias, and free data is riddled with it because delisted tickers are expensive to maintain. A momentum or value backtest on "current S&P 500 members since 2000" can overstate returns by several percent a year. There is no clean free fix; the honest workarounds are to use point-in-time constituent lists (usually paid), to backtest on liquid ETFs (which do survive as instruments), or to at least state the bias loudly in your results.

Trap 3, point-in-time vs restated data

Macro and fundamental data get revised. The US GDP or non-farm-payrolls number first released for a quarter is not the number FRED shows you today, it has been restated, sometimes substantially. If your backtest uses the current, restated value as of a historical date, you are trading on numbers that did not exist yet: a form of Look-Ahead Bias. The same applies to company fundamentals, reported earnings get amended, and databases often overwrite the original. Point-in-time data preserves what was actually known on each date. FRED exposes this via its ALFRED vintages; free fundamental data usually does not, so treat restated fundamentals as suspect for any signal timed near the release. When in doubt, lag the data generously past the real-world publication date.

Trap 4, rate limits and silent failures

Free APIs throttle you, and the ugliest failures are the silent ones: instead of an error, you get a truncated frame, an empty column, or a stale cached value. A yfinance call for 50 tickers may quietly return 47. Always validate the shape of what came back before trusting it, and cache aggressively so you pull each symbol once:

expected = {"SPY", "AAPL", "MSFT"}
got = set(df.columns.get_level_values(0))
missing = expected - got
assert not missing, f"silent partial download, missing: {missing}"

Space requests, add retries with backoff, and never put a live download inside a hot loop, pull to data/raw/ once, then work offline.

Trap 5, silent gaps and NaNs

Free feeds drop days. A holiday, a data outage, or a thinly traded name leaves holes, and a hole is dangerous precisely because it's silent: pct_change() will happily compute a two-day return across a one-day gap as if nothing is missing. Make gaps explicit by reindexing onto the calendar you expect:

import numpy as np, pandas as pd
cal = pd.bdate_range("2024-01-01", "2024-01-12")           # calendar you expect
have = cal.drop([pd.Timestamp("2024-01-04"), pd.Timestamp("2024-01-05")])
px = pd.Series(np.arange(len(have), dtype=float), index=have)

full = px.reindex(cal)                                     # holes become explicit NaN
print("rows before/after:", len(px), "/", len(full), "| NaNs exposed:", int(full.isna().sum()))
# -> rows before/after: 8 / 10 | NaNs exposed: 2

Now the gaps are visible and you can decide, deliberately, how to handle them. Do not reflexively forward-fill; that has its own lookahead hazards, covered in cleaning financial data.

Trap 6, timezone and session alignment

Combine a US equity feed, a European feed, and a 24/7 crypto feed and you will misalign bars unless timestamps are timezone-aware. A "daily close" means 16:00 America/New_York for US stocks, a different wall-clock time in London, and midnight-UTC (or exchange-local) for crypto. Naive timestamps silently paper over this:

import pandas as pd
ts_utc = pd.Timestamp("2024-03-11 20:00", tz="UTC")   # a US market close, in UTC
print("UTC 20:00 ->", ts_utc.tz_convert("America/New_York").strftime("%H:%M %Z"))
# -> UTC 20:00 -> 16:00 EDT   (the NY session close)

Always store timestamps as timezone-aware UTC, convert to the exchange's timezone only for session logic, and remember daylight-saving shifts the UTC offset twice a year. Mixing a crypto bar timestamped at UTC midnight with an equity bar timestamped at the NY close, without alignment, produces a lookahead that's nearly impossible to spot later.

Where this breaks

• You computed returns off raw close. Splits and dividends become fake ±50% jumps. Use adjusted/total-return series for returns; keep raw for traded prices.
• You backtested today's index members. That's Survivorship Bias baked in, state it or fix it with point-in-time universes.
• You used restated macro/fundamentals. Point-in-time or generous lags only; otherwise it's Look-Ahead Bias.
• You trusted the download blindly. Validate shape and completeness; free feeds fail silently.
• You ignored gaps and timezones. Reindex to expose holes; keep everything tz-aware.

Clean-looking data is not clean data. With prices pulled and stored, the next job is turning them into correct returns and aligned panels without leaking the future, the subject of cleaning financial data.