> People typically start with Excel and graduate to Pandas somewhere in the GB range. Pandas serves them well into the 10s of GBs range, and then they start hitting memory issues, slow computation, or become frustrated with Pandas’ baroque API.

Assumes that a project moves beyond 10s of GBs. I guess 99.9% of projects that import pandas fall well below this threshold.

▲

__mharrison__ 2 hours ago | parent | next [-]

Most folks just need to learn how to use pandas well and that will open enough doors. Then they can move to polars or duck if needed.

▲

__eddie__ 3 hours ago | parent | prev | next [-]

True, but also if Polars and DuckDB offer a similar experience with the ability to scale beyond that range, why not use them (for new projects)?

This isn't a call to arms to rewrite everything in the new shiny, just consider the new shiny for new shiny things

▲

appplication 3 hours ago | parent [-]

Not to mention less footguns. I used to spend my days unwinding bad habits DS pick up from years of panda abuse.

Credit to pandas for popularizing dataframes in Python, but polars and duckdb are objectively better APIs in addition to their implementation improvements. Agree that it’s time we let it go.

	▲	__eddie__ 3 hours ago \| parent [-]
		Yes, this was something I emphasised more in the slides / my actual talk. But it's just significantly easier to reason about the APIs for both DuckDB and Polars

▲

minimaxir 4 hours ago | parent | prev [-]

At my work I had convinced the ML pipeline engineers to switch from pandas to polars for even small ETL pipelines and there were notable performance gain with better CPU/memory utilization.

If a library is performant at large datasets, it is likely performant at small ones too.

▲

sjtrny 3 hours ago | parent [-]

I’m not disagreeing with that statement at all. You missed my point that there are thousands of people making small Python scripts for education and personal projects everyday. In those circumstances the performance concerns are irrelevant and the ergonomics of good pandas documentation and community knowledge make it a better choice.

	▲	minimaxir 3 hours ago \| parent [-]
		That inertia is not a good thing, and it's partially why there's stagnation in data science. Polars is more than mature enough in both documentation and resources for it to be a daily driver.