DuckDB vs pandas vs Polars on 50 Million Rows

Relative performance between DuckDB, pandas and Polars depends more on workload shape — particularly join count — than on raw dataset size, and peak memory differentiates them more sharply than wall time.

Six workloads run against the same data at 5M, 50M and 100M rows: full scan aggregation, group-by on a low-cardinality key, group-by on a high-cardinality key, two-table join, five-table join, and a row-wise window calculation.

Each configuration run five times after a warm-up; median reported. Peak resident memory sampled at 50ms intervals in a separate process.

Hardware held constant: 8-core laptop, 16GB RAM, NVMe storage. All engines read the same Parquet files.

Synthetic e-commerce data: orders, customers, products, suppliers, regions. 100M order rows at the largest size, generated with a fixed seed. Generator script in the repository.

pandas 2.x with PyArrow backend, Polars in lazy mode, DuckDB in-process. Each engine given its idiomatic implementation of each workload rather than a literal translation, since a literal translation would misrepresent all three.

Why memory belongs in the headline#

Almost every engine comparison reports time and omits memory. That is backwards for the audience most likely to read one: someone deciding whether a job will run on the machine they have.

A job that takes 24 seconds and 11GB and a job that takes 4 seconds and 2GB are not the same job with different speeds. On a 16GB laptop with a browser open, one of them completes and the other does not.

What this does not test#

Concurrency, larger-than-disk data, and distributed execution. Those are the situations where the answer changes, and they are also far less common than architecture diagrams imply.

Two findings, and only one of them is about speed.

The join count matters more than the row count. DuckDB's advantage over Polars is modest on single-table work and roughly 6× on the five-table join. That is the query planner: it reorders joins, and the dataframe libraries execute the order you wrote. As the number of tables grows, the gap between a good join order and a naive one grows with it.

Memory is the real constraint. The five-table join is the only workload where an engine failed outright, and it failed on memory, not time. DuckDB used a fifth of Polars' peak because it streams rather than materialising intermediates. On a 16GB laptop that is frequently the difference between a job running and not running — and it is invisible in every benchmark that reports only wall time.

Polars wins the row-wise window workload clearly, which is worth stating plainly: this is not a benchmark with a single winner. Expression-based row-wise transformation is genuinely its strength.

make benchmark regenerates the data and runs everything; roughly 40 minutes end to end at 50M rows. Environment pinned with a lock file, generator seeded, and raw timing output committed alongside the summary so the aggregation can be checked.