Exploratory Data Analysis

Before you fit a model, trade a strategy, or trust a number, you have to know what is actually in the data. A backtest built on a price series with a stray decimal error, a column of returns that are secretly percentages mixed with decimals, or a "missing" code stored as the number 999 will produce a confident, wrong answer. Exploratory data analysis (EDA) is the discipline of looking at the data first — with your eyes, not just a model — before you ask it to answer anything.

An analogy: walking the property before you buy it

A buyer doesn't judge a house from the listing photos and the asking price alone. They walk every room, open the cabinets, check the water pressure, and look for a suspicious patch on the ceiling. Most rooms are fine. The point of the walk-through is to find the one thing that isn't — the thing that changes the deal. EDA is that walk-through applied to a spreadsheet: you are not trying to prove the data is good, you are hunting for the specific way it might be bad.

What you actually do

There is no single formula for EDA — that is the point, it is a set of habits, not a computation. But the habits cluster into three questions, each with standard tools.

1. What does one variable look like on its own? Compute the center and spread — mean, median, standard deviation — and look at the shape with a histogram. If $\bar{x}$ (the sample mean) and the median disagree sharply, the distribution is skewed, and the mean is being pulled by a tail. In plain English: two numbers that "should" agree telling different stories is itself information.

$$\bar{x} = \frac{1}{n}\sum_{i=1}^{n} x_i$$

This just says: add up every observation $x_i$ and divide by how many there are, $n$. It is the arithmetic center of mass — and a single wild value can drag it far from where "most" of the data sits.

2. How do two variables relate? A scatter plot of $x$ against $y$, or the correlation coefficient, tells you whether they move together, and — more importantly for EDA — whether the relationship is a clean line or an artifact of a handful of extreme points.

3. Where does the data break the rules it should follow? Dates that go backward, prices that are exactly zero, a "volume" column with negative numbers, timestamps that repeat. These are not statistical questions; they are bookkeeping questions, and EDA is where you catch them, before a model quietly learns to exploit them.

Drag the mean and standard deviation above and watch how a histogram's shape changes with them — this is the picture you are checking your real data against every time you plot it.

Worked example 1: the mean that lied

You are handed 10 days of a stock's daily returns, in percent: $0.5, -0.3, 0.8, 0.2, -0.1, 0.6, -0.4, 0.3, 0.1, 45.0$. The last value looks like a typo — perhaps a return of $0.45\%$ that lost its decimal point.

Compute the mean including it:

$$\bar{x} = \frac{0.5 - 0.3 + 0.8 + 0.2 - 0.1 + 0.6 - 0.4 + 0.3 + 0.1 + 45.0}{10} = \frac{46.7}{10} = 4.67 .$$

A "typical" daily return of $4.67\%$ would be an extraordinary stock. Now drop the suspect value and recompute over the remaining 9 points:

$$\bar{x} = \frac{1.7}{9} = 0.189 .$$

One value moved the mean by a factor of about 25. That gap between $4.67$ and $0.189$ is exactly the kind of thing EDA is built to surface — not by a formal test, but by simply plotting the ten points and noticing that one of them sits nowhere near the others. A quick fix (drop it, cap it, or go verify the source) now saves a wrong Sharpe ratio later.

Worked example 2: correlation driven by one point

Five (x, y) pairs: $(1,1), (2,2), (3,3), (4,4), (20,1)$. The first four points lie on a perfect line $y = x$. Compute the correlation with all five points and it comes out close to zero — the last point, far from the rest on the x-axis but low on y, drags the relationship apart. Drop that one point and the correlation among the remaining four is exactly $1.0$. Nothing here required a formula to catch; a scatter plot showing four points on a line and one point isolated far to the right tells you immediately that the summary statistic is being distorted by a single observation, not describing the bulk of the data.

Drag the correlation slider above and watch the point cloud tighten or scatter — then imagine one point placed far outside the cloud, and notice how much it can move the fitted line and the reported $r^2$.

What this means in practice

• Before any backtest, plot the raw price and return series. A single spike, a flat line where prices should move, or a gap where data is missing will otherwise show up later as an inexplicable strategy edge or loss.
• Before any regression or factor model, check each input variable's histogram and its scatter against the target. A wrongly-scaled column (percent vs. decimal, or a currency column with a stray $ sign that failed to parse as a number) is invisible in a correlation matrix but obvious in a plot.
• Summary statistics are a starting point, not a verdict. Two datasets can share the same mean and standard deviation and look completely different — this is the entire lesson of skewness and kurtosis and the classic reason to plot, not just summarize.