Virtual Environments and Containers

Two strategies live on the same research server. One needs pandas 1.5 because its code relies on behavior that changed in later versions; the other needs pandas 2.2 to use a feature that doesn't exist in 1.5. Install one globally and the other breaks — there's only one system-wide copy of pandas a plain pip install can touch. This is the problem virtual environments and containers both solve, at two different levels of the stack.

A virtual environment (Python's venv, or tools built on it like conda) creates an isolated folder with its own copy of the Python interpreter and its own set of installed libraries, separate from the system Python and from any other project's environment. Activate strategy A's environment and pandas 1.5 is all that's visible; activate strategy B's and it's 2.2. Nothing about the underlying operating system, the version of Python itself, or system libraries like a C compiler is isolated — just the Python packages.

A container (Docker being the standard tool) goes a layer deeper: it packages not just the Python libraries but the operating system, system-level dependencies, and even the Python interpreter version itself into a single portable image. A container built on a research laptop runs identically on a cloud server or a colleague's machine, because it isn't relying on whatever happens to already be installed there — it brings its own complete environment with it, described in a Dockerfile.

Worked example. A team's live trading system needs a specific version of a C library for fast numerical linear algebra that isn't available through pip at all — it has to be installed at the operating-system level. A virtual environment can't help here, because it only isolates Python packages, not OS-level libraries; the underlying machine still needs the right system library installed globally, which risks conflicting with something else on that machine. A container sidesteps this entirely: the Dockerfile specifies the exact base OS image, installs the exact system library version, then installs the exact Python packages on top, and the resulting image is what actually runs in production. When the system is redeployed to a new server, there's nothing to configure — the container brings everything it needs.

What this means in practice

Virtual environments are the lightweight default for day-to-day research: fast to create, one per project, cheap to throw away and rebuild. Containers earn their extra complexity when a system needs to move between machines reliably — from a researcher's laptop to a shared backtest cluster to a live trading server — or when dependencies reach below the Python layer into the operating system itself. Using neither is what produces the classic "works on my machine" bug report, where a strategy runs fine for one person and crashes for another because their systems have quietly diverged.