Qm
Core

Chained Assignment and Copy-on-Write

The classic pandas trap of writing df[df.x > 0]["y"] = 1 and having it silently fail to update the original DataFrame, plus how pandas's newer copy-on-write mode changes and mostly fixes this behavior.

Prerequisites: Pandas DataFrame Internals, Index Alignment in Pandas

A line like df[df.pnl < 0]["flag"] = "loss" looks like it should set a column on every losing row, and it often runs without any error, yet the original df is left completely unchanged. This is chained assignment: the first bracket, df[df.pnl < 0], may return either a view into the original data or a brand-new copy, depending on internal details pandas doesn't guarantee to you, and the second bracket assignment then modifies whichever one it happened to get. When it gets a copy, your update vanishes into a temporary object that's immediately discarded, with no error raised, historically only a SettingWithCopyWarning, easy to miss in a noisy log.

The fix has always been to do the row and column selection in a single .loc call instead of two chained bracket operations: df.loc[df.pnl < 0, "flag"] = "loss" unambiguously operates on the original DataFrame in one step, with no intermediate object whose view-or-copy status is ambiguous.

Recent pandas versions introduce copy-on-write (default from pandas 3.0 onward), which changes the underlying mechanics: every selection now behaves as if it returned an independent copy, so chained assignment fails loudly and consistently, the assignment simply has no effect on the original, with a clear warning, rather than sometimes silently working and sometimes silently failing depending on internal memory layout. Copy-on-write removes the ambiguity, but .loc in one step is still the correct and unambiguous way to write the update.

Chained assignment (df[mask]["col"] = value) may silently fail to modify the original DataFrame because the first bracket can return a copy rather than a view; use a single .loc[mask, "col"] = value call instead, which is unambiguous under both classic and copy-on-write pandas.

Discussion

Sign in to join the discussion · reading is open to everyone

💡 Discussion rules

  1. Ask and answer about this concept. Off-topic gets removed.
  2. No homework dumps. Show what you tried first.
  3. Corrections are welcome. Cite a source when you claim an error.

Loading discussion…

Related concepts

Practice in interviews

Further reading

  • pandas documentation, 'Copy-on-Write'
ShareTwitterLinkedIn