Concept Activation Vectors and TCAV

Most explanation methods answer "which input features mattered for this prediction?" Concept Activation Vectors (CAVs) ask a different, arguably more useful question: does this network's internal representation respond to a human concept at all, and how much does that concept push its predictions? "Earnings surprise," "risk-off regime," or "momentum crash" aren't single input columns — they're patterns a network might learn internally without any feature being explicitly labeled that way, and TCAV (Testing with CAVs) is the method for detecting whether it has.

How a CAV is built

Pick a concept and gather two sets of examples: some clearly exhibiting it (say, days classified by a human as "risk-off," meaning falling equities, rising bonds, rising volatility) and some that don't (ordinary days). Feed both sets through the network and pull out the internal activations at some chosen layer. Train a simple linear classifier — logistic regression works fine — to separate the "concept" activations from the "non-concept" activations. The vector perpendicular to that classifier's decision boundary, pointing toward the concept examples, is the Concept Activation Vector: a direction in the network's internal representation space that corresponds to "risk-off-ness."

TCAV then measures, for a target prediction (say, "reduce leverage"), how often nudging an input's internal activation slightly along the CAV direction increases the network's predicted probability of that output. This is the TCAV score: the fraction of examples where the concept direction and the prediction move together, giving a number between 0 and 1 for how much that concept drives that prediction.

Worked example

A firm trains a network to predict whether to cut position size, using only raw price, volume, and order-book features — no feature is explicitly labeled "regime." Researchers hand-label 200 historical days as risk-off and 200 as normal, extract activations at the network's second-to-last layer for both sets, and fit a logistic classifier that separates them with 92% accuracy, giving a usable CAV. Running TCAV against the "cut position size" output over a held-out year, the concept direction aligns with an increased cut-size probability on 78% of days — a TCAV score of 0.78 — evidence the network has implicitly learned something resembling a risk-off concept and leans on it heavily, even though "risk-off" was never an input feature.

What this means in practice

TCAV lets a desk audit whether a model's internal reasoning tracks something economically sensible, like regime awareness, without needing that concept to have been an explicit input — useful both for building trust and for catching cases where a model has instead keyed on a spurious concept, like day-of-week, that happens to correlate with the labels. The result depends heavily on choosing good, clean concept examples and the right layer; a sloppily labeled concept set produces a CAV that measures something other than what you intended.