Web Scraping for Signals
Web scraping turns publicly visible web pages — prices, job listings, product reviews — into structured, trackable data over time, but the legal and practical fragility of scraping is as much a part of the strategy as the signal itself.
Prerequisites: Sourcing and Vetting Alternative Data
Company websites are full of information that is technically public but never packaged into a data feed — a retailer's current prices, a company's open job listings, the number of reviews on a product page. Web scraping is writing automated programs that visit these pages repeatedly and pull that information into a structured dataset, turning a page that changes silently into a time series that can be tracked and analyzed like any other data source.
Web scraping converts publicly visible but unstructured web pages into a structured, repeatable dataset by programmatically visiting them on a schedule and extracting the same fields each time — the value is entirely in doing this consistently enough, and for long enough, to see a trend.
What gets scraped, and why it's useful
Common targets include retailer prices (to track discounting and competitive pressure), job postings (to gauge a company's hiring pace as a proxy for growth plans), and product review counts or ratings (to gauge demand momentum for a specific product line before sales figures are reported). None of these is proprietary data anyone is selling — it is sitting on a public webpage — but nobody manually checks thousands of product pages every day, so the act of automating and structuring the collection is itself the value being created.
Worked example
An analyst scrapes a retailer's website nightly and tracks the listed price of its 50 best-selling products. Over six weeks, the average discount off list price widens from 5% to 18% across the tracked basket. That widening discount, visible weeks before the company's next earnings call, suggests the company is discounting more aggressively to move inventory — a signal consistent with softer-than-expected demand, well ahead of any official disclosure.
What this means in practice
Scraping is fragile in ways that vendor-supplied data usually isn't: a website redesign can silently break the extraction code overnight, and the target site can change its terms of service or add technical blocks aimed specifically at automated visitors. It also carries real legal exposure — scraping is not automatically legal just because a page is publicly viewable, and courts have reached different conclusions depending on the site's terms of service, whether login is required, and how the scraped data is used. A serious scraping-based strategy budgets for continuous engineering maintenance and involves legal review before deployment, not just a one-time script.
Treat every scraper as a data pipeline that will break — build monitoring that flags when a scrape returns an unexpectedly empty or malformed result, so a silent website redesign doesn't quietly turn six months of "data" into six months of stale garbage.
Related concepts
Practice in interviews
Further reading
- Kolanovic and Krishnamachari, 'Big Data and AI Strategies' (JPMorgan), section on web-scraped alt data