Strongly disagree with the author.

For dumb simple select group by OLAP queries on medium data ? Sure clickhouse local or duckdb works perfectly well.

But if you need to construct dataset ? Or process existing dataset, do heavy filtering, transformation, reshaping, splitting? The proper ETL work, then pandas is really the perfect use case.

And pandas can work with small memory footprint as well, its actually trivial to do that, plus there are libraries like Modin that are upgrades over pandas with pandas api

Pandas is the swiss knife tool of data science that lets you do anything with the data and it integrates well with ML libraries

▲

minimaxir 3 hours ago | parent | next [-]

It's subjective, but IMO polars's APIs are better and than pandas's for ETL. Notably, pandas ported over polars's use of .col() for the 3.0.0 release: https://pandas.pydata.org/docs/whatsnew/v3.0.0.html#initial-...

Also, pandas's group_by() and window function pipelines made me want to tear my hair out while both are easy in polars.

▲

__mharrison__ 2 hours ago | parent | prev [-]

Use pandas if you want advanced analytics, visualization, or ml.

Use SQL if you need to move data around.

▲

bijowo1676 2 hours ago | parent [-]

How would you use sql to move data from three excel files (some of them may have arbitrary number of worksheets), pack of json files, two csv files, one mysql, one postgres db, some parquets, some of them in S3) ???

And in the end save processed data in aws s3 in another format like iceberg or whatever

In pandas a lot of these are one-liners that are impossible in sql (depends on what sql engine you have?)

	▲	__mharrison__ an hour ago \| parent [-]
		Those are generally rules of thumbs for data pipelines. (Plus duckdb can read Excel these days I think).