Metrics, Dashboards and Alerting

A live trading system can generate thousands of log lines an hour, but nobody is going to sit and read them as they happen. What a team actually needs at a glance, in real time, is a much smaller set of numbers: is P&L within a normal range, is order latency where it should be, is the data feed still updating, are there more failed orders than usual in the last five minutes. That smaller set of numbers, computed continuously and tracked over time, is a system's metrics — things like current position, rolling P&L, orders-per-minute, data feed staleness, or error rate. Metrics are what logs get distilled into when the goal is monitoring the system's health right now, rather than investigating one specific past event.

A dashboard is a visual arrangement of those metrics — line charts of P&L over the day, a gauge of current exposure against limits, a counter of recent errors — designed so a person can glance at one screen and judge "is everything normal" without reading a single log line. Alerting automates the part a dashboard still requires a human to do (actively look at it): a rule that watches a metric continuously and fires a notification — a page, a Slack message, an email — the moment it crosses a defined threshold, so a problem gets noticed within seconds even if nobody happened to be looking at the dashboard at that moment.

Worked example. A team tracks the metric orders_rejected_per_minute on a live dashboard, normally hovering near zero. A broker connectivity issue causes rejections to spike to 40 per minute. If the team relies purely on the dashboard, the spike is only caught whenever someone next glances at that chart — potentially minutes or hours later. Instead, an alert rule is configured: if orders_rejected_per_minute exceeds 10 for more than 60 seconds, page the on-call engineer immediately. The spike triggers the alert within a minute of starting, the engineer is paged, and the connectivity issue is investigated and the affected strategy paused well before it has accumulated a meaningful number of failed trades — a response time no amount of dashboard-watching alone would reliably guarantee.

What this means in practice

Metrics, dashboards, and alerting form a hierarchy of urgency: metrics are the raw numbers, dashboards make them glanceable for routine checks, and alerts make sure the ones that actually require immediate action reach a human without anyone needing to be watching. A common mistake is over-alerting — setting thresholds so sensitive that alerts fire constantly for non-issues, which trains people to ignore them, so that the one alert that matters gets dismissed along with the noise. Good alerting is deliberately sparse: it fires on the handful of conditions that genuinely warrant waking someone up, and everything else stays on the dashboard for the next routine check.