Remix.run Logo
viccis 3 hours ago

Polars seems nice but in my experience using it, the "lazy" APIs would still immediately materialize a ton of stuff in memory and had very spotty support on what data formats and storage integrations were possible with scan_* functions (though that was half a year ago and the support is slowly improving). It's frustrating, I mean really frustrating, to think I could solve a lot of my "scan through heinous amounts of data without any memory hungry things like window aggregations without blowing out my memory" with Polars and then watch my scan_thisorthat() call result in instant memory usage ballooning.

DuckDB on the other hand is wonderful and truly doesn't use any more memory than it really needs to.

__mharrison__ 2 hours ago | parent [-]

Did you ever report these issues to polars developers?

I reported many bugs to both pandas and polars over the years and both teams tend to address relatively big ones. (Still have some outstanding pandas bugs that I think are a big deal but the devs disagree.)