DuckDB vs pandas vs Polars: When Each One Wins
The benchmark posts all measure the same thing — a group-by aggregation on a wide table — and then people generalise from it to workloads that behave nothing like it. Here is a decision guide organised by what you are doing instead.
Pick by workload shape#
| Your workload | Use | Why |
|---|---|---|
| Multi-table joins and aggregations | DuckDB | A real query planner; joins are its core competence |
| Row-wise transformations over one table | Polars | Lazy expressions with excellent single-table throughput |
| Exploratory work in a notebook | pandas | The ecosystem is the feature; everything integrates with it |
| Data larger than memory | DuckDB | Out-of-core execution built in |
| Feeding scikit-learn or similar | pandas or Polars | Whatever your ML library accepts natively |
| A pipeline someone else maintains | DuckDB | SQL is the most widely readable option |
The part benchmarks miss#
Joins. DuckDB's advantage grows with the number of tables, because it has a cost-based optimiser and the dataframe libraries execute your join order as written. On a five-table join, the difference is often an order of magnitude — not because of raw speed, but because the planner picked a better order than you did.
Memory. pandas comfortably uses 5–10× the file size in RAM. Polars is much better. DuckDB streams. On a 16GB laptop this is frequently the deciding factor, and it never shows up in a timing chart.
Maintainability. SQL is readable by more people in most organisations than method-chained dataframe code. That is a real engineering consideration, not a soft one.
They compose#
The most useful thing to know is that this is rarely an either/or:
mixing.pypythonimport duckdb
import polars as pl
# DuckDB does the heavy multi-table work.
df = duckdb.sql("""
SELECT c.segment, date_trunc('month', o.order_date) AS month,
sum(o.amount) AS revenue
FROM 'orders/*.parquet' o
JOIN 'customers.parquet' c USING (customer_id)
GROUP BY 1, 2
""").pl()
# Polars does the row-wise shaping on the much smaller result.
result = df.with_columns(
(pl.col("revenue") / pl.col("revenue").sum().over("month")).alias("share")
)
Zero-copy through Arrow means moving between them is nearly free. The right answer is usually "the one that suits this step", not "the one that won the benchmark".
A default worth having#
If you have no strong reason to choose otherwise: DuckDB for anything involving joins or files, Polars for single-table transformation, pandas when a library you depend on requires it. That default is right more often than any benchmark-derived ranking.
Get new projects, datasets, notebooks and system builds.
One email a week. Source code and files included. No fluff, no recycled LinkedIn posts.