| ▲ | latent-person 2 hours ago |
| The problem is a small change in the question can force you to make huge changes in the code in pandas.
I recently gave some examples in my blog [1]. E.g. compare the last two code blocks, where the small change is just that the median is taken within countries. This requires several line changes in pandas. [1]https://bjarkehautop.github.io/Website/blog/data-wrangling-t... |
|
| ▲ | anakaine an hour ago | parent | next [-] |
| Personally, I think the inclusion of lambda functions in the pandas code shows that the approach isnt ideal to begin with. Fine on smaller datasets, but typically becomes a slow and cumbersome overhead when interacting with larger datasets. More verbose per line vector operations are generally orders of magnitude faster, and become more so as the data grows. They also tend to be easier to read / identify what is going on or what the intent is for people unfamiliar with the code base. |
| |
| ▲ | __mharrison__ an hour ago | parent [-] | | This is a common complaint I get all the time (heard it this week while teaching pandas). I compare this to whitespace indentation in Python. Lots of folks complain about it before using it. After they use it it is a non issue. If it really is an issue (and it generally isn't a cause of vectorization removal when used correctly) and you can't get over the syntactic noise if the lambda, pandas 3 introduced pd.col (that work in most (I filed a big about some exceptions) places when you'd use lambda). | | |
| ▲ | anakaine an hour ago | parent [-] | | Ive seen it in such a wide variety of scenarios where it causes vectorisation removal because of its implementation that I dont reach for it by default any more. I've a number of devs around me who have similar opinions. Each know how to use it, but you dont always write the code you're reviewing or optimising. |
|
|
|
| ▲ | __mharrison__ an hour ago | parent | prev [-] |
| I looked at your code very quickly, but it looks like you need to use .filter after a .groupby... |
| |
| ▲ | latent-person an hour ago | parent [-] | | The text right above the code says why you can't... edit: Let me clarify. From the blog-post: > since a `DataFrameGroupBy` object doesn’t have a `.query()` or boolean-indexing shortcut of its own, so filtering within groups needs `.apply()` again, and the surrounding pipeline has to be rebuilt around it: Hence you really do need one of the versions of the code I gave. You can't do the naive approach with just `.groupby().filter(lambda: )`, since you need a row-wise decision. | | |
| ▲ | __mharrison__ an hour ago | parent [-] | | I'm confused, you can use filter after a groupby in pandas... It's late here, I'm going to bed, perhaps I'll write the code tomorrow when I'm at my laptop and not on my phone. | | |
|
|
