DuckDB 2.0 on a CSV too big for Excel: 3 million rows queried in 0.2 seconds
We ran DuckDB 2.0.0 on the day after release against a synthetic 3,000,000-row, 123 MB CSV, almost three times Excel's 1,048,576-row limit. A GROUP BY took 0.17 to 0.22 seconds, against 4.04 seconds for a plain Python csv loop that returned the same totals to the cent. One 2.0 API change broke our script.
By the RanThis team, written with AI4 min read
Why trust this: every result, number and screenshot comes from a real run on our own accounts, not from the AI. How we test
DuckDB 2.0 read a 3-million-row CSV (123 MB) and grouped it in about 0.2 seconds on our PC. That file has almost three times the rows Excel will open. A plain Python csv loop doing the same grouping took 4.0 seconds. You write the question in SQL, no import wizard, no server.
We tested on 22 Oct 2026, one day after the release, with duckdb 2.0.0 from PyPI (Python 3.13, a 12-thread desktop, 7 GB of RAM in WSL). The data is synthetic: 3,000,000 fake orders generated with a fixed random seed. We did not open this file in Excel, so everything about Excel below is its published limit, not something we measured.
At a glance
Key results
- Query time:
GROUP BYregion and category over 3,000,000 rows, three runs: 0.22 s, 0.18 s, 0.17 s. A barecount(*)on the same file: 0.13 s. - Baseline: a plain Python loop with the
csvmodule computing the same sums took 4.04 s, about 20× slower. It returned the same top group (North / Phone) with the same total, 228,363,763.10 to the cent, which is how we know the SQL did the right thing. - File size: 122.8 MB, six columns. The file never has to fit in a spreadsheet; DuckDB reads it straight from disk.
- Breaking change we hit: in 2.0 a query object no longer has
fetchone(). Our script from the 1.x habit crashed with an error telling us to userel.limit(1).fetchall().
Hands-on test
Setup
- Release: DuckDB 2.0.0 LTS "Cinnamon Teal", announced on 21 Oct 2026. The team says 2.0 is supported until at least October 2027, and 1.4 until late November 2026 (announcement).
- Install:
uv run --with duckdb python script.py, nothing else. The Python client was already available on release day; the announcement says some other clients (Go, R, Java) can take a few days. - Data:
orders.csvwithorder_id, region, category, qty, unit_price, order_date, 3,000,000 rows, generated by a script with seed 7. The script is in our repo next to this article's source. - Query:
SELECT region, category, sum(qty * unit_price) AS revenue, count(*) AS n
FROM read_csv('orders.csv')
GROUP BY ALL
ORDER BY revenue DESC
LIMIT 3;
read_csv works out the column types by itself, and GROUP BY ALL groups by every column you did not aggregate.
Why a CSV like this is a problem in Excel
Excel's worksheet limit is 1,048,576 rows (Microsoft: Excel specifications and limits). Our file has 3,000,000 data rows, so Excel cannot show all of it in one sheet. We did not run this on Excel; we only compare the row counts. If you have a CSV exported from a shop, a bank or a log system and it passes that limit, the choices are usually to split it by hand or to query it where it sits.
What worked
What worked
- Zero setup for a one-off question. One line of Python or one
duckdbcommand, and the file stays a file. No database to create. - Speed without tuning. We did not add indexes or convert the CSV to Parquet first. All three runs finished under a quarter of a second.
- Answers match. The SQL result equals the independent Python loop to the cent.
Where it fell short
What didn't work
- API change in 2.0. Code that called
.fetchone()on a query object breaks. Use.fetchall()or.limit(1).fetchall(). If you have older DuckDB snippets from 1.x, expect to touch them. - Not a spreadsheet. You get rows back, not cells you can edit. To hand the result to someone who lives in Excel you still export a small result (a few hundred rows), which is the point, but it is an extra step.
- One machine, one file shape. Six plain columns, no quoted newlines, no messy encodings. Real exports can be uglier, and we did not test them. We also did not test pandas or Polars, so "20× faster" is against a pure-Python loop only.
The 2.0 features we did not use
The release also adds a VARIANT type for semi-structured data, triggers, a Quack client-server protocol with a CONNECT statement, asynchronous I/O for Parquet and CSV on remote storage, and an "agent mode" in the CLI that prints compact Markdown tables when a coding agent drives it. We did not test any of these. Per the announcement, agent mode cut the DuckDB output the model had to read by 59% in the team's own TPC-H experiment.
Our verdict
If your question is "sum, count or filter this huge CSV" and you can type basic SQL, DuckDB 2.0 answered in a fraction of a second on a laptop-class machine where a hand-written Python loop needed four. Check the Excel row limit against your file first; if you are under it, Excel is fine and you do not need this. If you have older DuckDB code, search it for fetchone before upgrading.
