What's wrong with Pandas?

crystal_revenge an hour ago | parent | next [-]

Pandas is generally awful unless you're just living in a notebook (and even then it's probably least favorite implementation of the 'data frame' concept).

Since Pandas lacks Polars' concept of an Expression, it's actually quite challenging to programmatically interact with non-trivial Pandas queries. In Polars the query logic can be entirely independent of the data frame while still referencing specific columns of the data frame. This makes Polars data frames work much more naturally with typical programming abstractions.

Pandas multi-index is a bad idea in nearly all contexts other than it's original use case: financial time series (and I'll admit, if you're working with purely financial time series, then Pandas feels much better). Sufficiently large Pandas code bases are littered with seemingly arbitrary uses of 'reset_index', there are many times where multi-index will create bugs, and, most important, I've never seen any non-financial scenario where anyone has ever used Multi-index to their advantage.

Finally Pandas is slow, which is honestly the least priority for me personally, but using Polars is so refreshing.

What other data frames have you used? Having used R's native dataframes extensively (the way they make use of indexing is so much nicer) in addition to Polars both are drastically preferable to Pandas. My experience is that most people use Pandas because it has been the only data frame implementation in Python. But personally I'd rather just not use data frames if I'm forced to used Pandas. Could you expand on what you like about Pandas over other data frames models you've worked with?

▲

clickety_clack 8 hours ago | parent | prev | next [-]

I probably wouldn’t rewrite an entire data science stack that used pandas, but most people would use polars if starting a new project today.

▲

biofox 7 hours ago | parent | next [-]

R and Matlab workflows have been fairly stable for the past decade. Why is the Python ecosystem so... unstable? It puts me off investing any time in it.

	▲	clickety_clack 6 hours ago \| parent \| next [-]
		The R ecosystem has had a similar evolution with the tidyverse, it was just a little further ago. As for Matlab, I initially learned statistical programming with it a long time ago, but I’m not sure I’ve ever seen it in the wild. I don’t know what’s going on there. I’m actually quite partial to R myself, and I used to use it extensively back when quick analysis was more valuable to my career. Things have probably progressed, but I dropped it in favor of python because python can integrate into production systems whereas R was (and maybe still is) geared towards writing reports. One of the best things to happen recently in data science is the plotnine library, bringing the grammar of graphics to python imho. The fact is that today, if you want career opportunities as a data scientist, you need to be fluent in python.
	▲	crystal_revenge an hour ago \| parent \| prev \| next [-]
		I love R, but how can you make that claim when R uses three distinct object-oriented systems all at the same time? R might seem stable only because it carries along with it 50 years of history of programming languages (part of it's charm, where else can you see the generic function approach to OOP in a language that's still evolving?) Finally, as someone who wrote a lot of R pre-tidyverse, I've seen the entire ecosystem radically change over my career.
	▲	rbartelme 6 hours ago \| parent \| prev [-]
		Outside bioconductor or the tidyverse in R can be just as unstable due to CRAN's package requirements.

▲

7 hours ago | parent | prev [-]

[deleted]

▲

amelius 6 hours ago | parent | prev [-]

Pandas turns 10x developers with a lust for life into 0.1x developers with grey hairs.

	▲	cbare 2 hours ago \| parent [-]
		Ha, I think that happens regardless of the tech you use. Just blame time.