Recently, the popularity of the Polars library, which offers significant performance advantages compared to pandas, has been steadily increasing. However, in order to understand how it performs against DuckDB, I believe it is necessary to conduct a small benchmark comparison.
In this benchmark, which I ran on my personal machine, I think it is more meaningful to focus on the ratio between the execution times rather than the absolute durations themselves. First, let me explain how I designed the comparison.
I decided to work with a CSV file of approximately 113 million rows available on Kaggle, the link to which is provided below. The file size is approximately 4.5 GB. The goal was to first read this CSV file, then group the data by a specific column, and finally sort the results to ensure a fair comparison.
In Polars, since grouping requires sorting beforehand rather than sorting after grouping, I could not implement the logic in a way that fully resembles writing pure SQL. However, on the other hand, since grouping operations in databases inherently involve sorting, I do not think there is a fundamental difference between their execution plans.
Another point is that the CSV file I used does not contain a header row. As a result, Polars and DuckDB apply different column naming methodologies. Therefore, you will notice that the column names used for grouping differ between the two implementations in the code.
The code is as follows:
import duckdb as dbimport polars as plimport datetimedef polars_job(): now=datetime.datetime.now() print("Polars Timing (read text data)") print("===================================") print(f'Start time: {now.strftime("%H:%M:%S")}') df = pl.read_csv( "custom_1988_2020.csv", has_header=False, columns=[ 'column_1','column_2','column_3','column_4', 'column_5','column_6','column_7','column_8' ] ) print(df.sort("column_3").group_by("column_3").len().head(10)) now = datetime.datetime.now() print(f'End time: {now.strftime("%H:%M:%S")}') del df print("***********************************")def duckdb_job(): conn = db.connect() now=datetime.datetime.now() print("Duckdb Timing (read text data)") print("===================================") print(f'Start time: {now.strftime("%H:%M:%S")}') df = conn.execute(""" select "column2", count(*) from read_csv('custom_1988_2020.csv', AUTO_DETECT=TRUE) group by "column2" order by "column2" """).df() print(df.head(10)) now = datetime.datetime.now() print(f'End time: {now.strftime("%H:%M:%S")}') conn.close() print("***********************************")if __name__ == '__main__': print(f"Polars version: {pl.__version__}") print(f"DuckDB version: {db.__version__}") print("***********************************") polars_job() duckdb_job()import duckdb as dbimport polars as plimport datetimedef polars_job(): now=datetime.datetime.now() print("Polars Timing (read text data)") print("===================================") print(f'Start time: {now.strftime("%H:%M:%S")}') df = pl.read_csv( "custom_1988_2020.csv", has_header=False, columns=[ 'column_1','column_2','column_3','column_4', 'column_5','column_6','column_7','column_8' ] ) print(df.sort("column_3").group_by("column_3").len().head(10)) now = datetime.datetime.now() print(f'End time: {now.strftime("%H:%M:%S")}') del df print("***********************************")def duckdb_job(): conn = db.connect() now=datetime.datetime.now() print("Duckdb Timing (read text data)") print("===================================") print(f'Start time: {now.strftime("%H:%M:%S")}') df = conn.execute(""" select "column2", count(*) from read_csv('custom_1988_2020.csv', AUTO_DETECT=TRUE) group by "column2" order by "column2" """).df() print(df.head(10)) now = datetime.datetime.now() print(f'End time: {now.strftime("%H:%M:%S")}') conn.close() print("***********************************")if __name__ == '__main__': print(f"Polars version: {pl.__version__}") print(f"DuckDB version: {db.__version__}") print("***********************************") polars_job() duckdb_job()import duckdb as dbimport polars as plimport datetimedef polars_job(): now=datetime.datetime.now() print("Polars Timing (read text data)") print("===================================") print(f'Start time: {now.strftime("%H:%M:%S")}') df = pl.read_csv( "custom_1988_2020.csv", has_header=False, columns=[ 'column_1','column_2','column_3','column_4', 'column_5','column_6','column_7','column_8' ] ) print(df.sort("column_3").group_by("column_3").len().head(10)) now = datetime.datetime.now() print(f'End time: {now.strftime("%H:%M:%S")}') del df print("***********************************")def duckdb_job(): conn = db.connect() now=datetime.datetime.now() print("Duckdb Timing (read text data)") print("===================================") print(f'Start time: {now.strftime("%H:%M:%S")}') df = conn.execute(""" select "column2", count(*) from read_csv('custom_1988_2020.csv', AUTO_DETECT=TRUE) group by "column2" order by "column2" """).df() print(df.head(10)) now = datetime.datetime.now() print(f'End time: {now.strftime("%H:%M:%S")}') conn.close() print("***********************************")if __name__ == '__main__': print(f"Polars version: {pl.__version__}") print(f"DuckDB version: {db.__version__}") print("***********************************") polars_job() duckdb_job()The benchmark was conducted using Polars version 0.20.6 and DuckDB version 0.9.2.
Polars indeed delivered significantly better performance compared to what would be expected from pandas. However, grouping and sorting 130 million rows by a specific column still took slightly over one minute. While the API differs somewhat from pandas, it should be very easy to adapt for those already familiar with pandas.
Based on a reader’s suggestion, I made the following change and observed that the execution time on the Polars side dropped to around 40 seconds. However, this change does not significantly alter my overall conclusions.
- print(df.sort("column_3").group_by("column_3").len().head(10))+ print(df.group_by("column_3").len().sort("column_3").head(10))- print(df.sort("column_3").group_by("column_3").len().head(10))+ print(df.group_by("column_3").len().sort("column_3").head(10))- print(df.sort("column_3").group_by("column_3").len().head(10))+ print(df.group_by("column_3").len().sort("column_3").head(10))
DuckDB, on the other hand, completed the same operation in approximately 12 seconds. Additionally, since Polars requires writing code while DuckDB allows querying directly using SQL, DuckDB felt much more practical to me. Especially for users with SQL knowledge, I believe DuckDB is both a very fast and a much more convenient alternative.
You can run the above code and use the dataset provided at the link below to test the results in your own environment:
https://www.kaggle.com/datasets/zanjibar/100-million-data-csv/data
