Remix.run Logo
Remix clone Hacker News
new | show | ask | jobsGithub
▲ Pre-Release of Polars 2.0(pola.rs)
221 points by komape 5 hours ago | 63 comments
▲benrutter 4 hours ago | parent | next [-]

> We don’t aim to make a big feature release of Polars 2.0. In fact we hope it to be a boring experience for you. The reason we bump this major version is that we can get rid of design decisions made in the past that currently block us and then we want to change defaults to more sensible settings that will benefit a greater audience

I know this take reveals me as a very dull person, but I love seeing projects take semver seriously like this! Version bumps should really be about removing deprecated cruft rather than shiny new features.

I've used polars for a while now, and their focus on stability was a big part if convincing me to make the jump initially!

▲altern8 42 minutes ago | parent | next [-]

Aren't major versions supposed to indicate breaking changes..?

That's how I thought semantic versioning worked

▲suddenlybananas 40 minutes ago | parent [-]

>The reason we bump this major version is that we can get rid of design decisions made in the past that currently block us and then we want to change defaults to more sensible settings that will benefit a greater audience

I don't know how to read this sentence other than "there are breaking changes we want to make"

▲magicmicah85 10 minutes ago | parent | next [-]

The migration guide does say there are breaking changes, but the interpretation I have is "this won't have new features but allows us to develop new features".

▲altern8 12 minutes ago | parent | prev [-]

I see. Just making sure I had it right :-)

▲nicce 3 hours ago | parent | prev | next [-]

> Version bumps should really be about removing deprecated cruft rather than shiny new features.

Can there be deprecated cruft without new features? :-D

▲SkiFire13 43 minutes ago | parent | next [-]

Yes, the features don't need to be added immediately.

▲kvdveer an hour ago | parent | prev [-]

Ideally: no.

All new shiny new features shouldn't have waited for the (N+1).0 version, they should already have been part of the (N).(M) version.

In practice, the removing the deprecated cruft will remove blockers for some new features, but that should be rare.

▲dist-epoch 3 hours ago | parent | prev | next [-]

That being said Polars is one of the few Python libraries from the hundreds I use that I need to read the notes of every minor release (eg 1.44 -> 1.45), because they tend to frequently deprecate, remove or change features.

▲duskdozer an hour ago | parent [-]

It sounds like they should be on a version much higher than 2.x then.

▲Bluestein 4 hours ago | parent | prev [-]

"Tranquil development" (vs. "hype-driven shipping") :)

▲trombonechamp 33 minutes ago | parent | prev | next [-]

Is there a reason besides performance that maintain_order=False by default? I ask because polars is used in many scientific data analysis pipelines, and non-deterministic behaviour is a well-documented source of bugs in scientific computing (e.g. https://pmc.ncbi.nlm.nih.gov/articles/PMC6919963/). The new default requires users to keep the implementation details of the API in their head while determining whether code is correct or not. This is tricky with scientific computing because the correct answer is not known in advance, so bugs can slide by and silently give incorrect results.

▲Kydlaw an hour ago | parent | prev | next [-]

Happy to see activity around Polars. This has been my go-to library for data processing due to the enhanced ergonomics compared to Pandas and SQL.

But they were a bit quiet lately, and I started looking more and more into DuckDB recently… until the recent acquisition of DuckLab by AWS

▲f311a an hour ago | parent [-]

I've been using clickhouse-local for quite some time, instead of DuckDB. There is also chDB.

After using pandas for 10 years, I favor SQL now, for some reason.

▲efromvt 39 minutes ago | parent | next [-]

As a huge duckdb fan, I'd love to see chDB to get proper windows support - that would make it real competition (having WASM coverage is already a big step) which would be good for the space as a whole.

▲mihaelm an hour ago | parent | prev [-]

Maybe because it's like a swiss army knife for data work, regardless of whether you need it for OLTP or OLAP workloads. Having different SQL dialects is a bit annoying, but the base is the same more or less, so switching doesn't come at too big of a cost.

▲bobson_dugnutt5 3 hours ago | parent | prev | next [-]

I love polars. Did a lot of evangelizing in work to get people to give up pandas in favor of it.

▲anotherpaul 3 hours ago | parent | next [-]

I gave up pandas in favor of polars after someone at work did the same and I am very happy with it. Pandas API is just so much worse and much slower.

▲duskdozer 42 minutes ago | parent | prev | next [-]

I guess I am a casual pandas user only. Reading a guide on migrating/differences, it's hard to see why polars would be obviously better.

▲bobson_dugnutt5 24 minutes ago | parent [-]

Here are a couple reasons:

- much faster, multithreaded by default. Read in a big csv with it and see how it feels.

- no index/MultiIndex. Pandas special treatment of index always felt like more trouble than it was worth, so no need to reset_index() everywhere.

- expressions are very portable. At first using pl.col everywhere feels like a bit much, but you can define them anywhere and then apply them to a dataframe whenever you want.

- once internalized, the syntax makes much more sense and is far more consistent compared to pandas.

Of course all depends on what your use cases are. If performance is important then I'd strongly recommend trying it out. If you just use it to have a look at the odd dataframe, maybe not worth your time as much

▲latexr 27 minutes ago | parent | prev | next [-]

Taken out of context, your post looks like a conservationist who got fed up with pandas being a flagship species and made it their lifelong mission to replace them with polar bears.

This is not a criticism. As someone who doesn’t use Python, I simply found it amusing.

▲blitzar 16 minutes ago | parent [-]

You should learn Boa constrictor instead of Python.

▲mgaunard 3 hours ago | parent | prev [-]

Both have terrible syntax that make SQL look like the most readable thing ever.

▲condwanaland 3 hours ago | parent | next [-]

Could not agree less. Ive always found SQL an unreadable mess but tools like polars and dplyr are such elegant ways to manipulate data.

Pandas is a mess though.

▲world2vec 2 hours ago | parent [-]

There's no way SQL is more unreadable than polars. IMO it's the other way around.

▲benrutter an hour ago | parent [-]

> There's no way SQL is more unreadable than polars. IMO it's the other way around.

I think on basic queries, SQL is really nice, but when stuff gets more complex, with a bunch of CTEs, let alone functions requiring loops, it becomes pretty obtuse.

▲aquafox 3 hours ago | parent | prev | next [-]

Coming from an R/dplyr background, I agree. Compare

df.select(

  pl.col("x"),
  (pl.col("w")/pl.col("z")).alias("y")
)

with

df |> select(x, y = w/z)

▲countrymile 3 minutes ago | parent | next [-]

Polars is a world away from pandas, but I feel that dplyr still offers the most simple and understandable introduction to data analysis for the beginner. The above is a good example of this.

▲orlp 2 hours ago | parent | prev | next [-]

    from polars import col as C

    df.select(C.x, y = C.w / C.z)
▲dkga 43 minutes ago | parent [-]

Still, it’s a very good approximation but still an approximation to the more ergonomic and expressive tidyverse syntax

▲bobson_dugnutt5 3 hours ago | parent | prev | next [-]

Fair point, but you can do something like

`df.select("x", y=pl.col.w/pl.col.z)`

▲jcattle 2 hours ago | parent | prev [-]

R really is/was the superior traditional data science language. Python ecosystem is slowly catching up though.

ggplot vs matplotlib

dplyr vs pandas

And I loved that everything in RStudio was so easily inspectable. Have a huge dataframe? Just look at it right in your IDE.

▲vovavili an hour ago | parent [-]

Altair and Positron should be just as good for your Polars @ Python needs. With software like Marimo notebooks and VegaFusion, Polars/Python experience starts beating R by quite a substantial margin.

▲bobson_dugnutt5 3 hours ago | parent | prev | next [-]

What is it about polars syntax you don't like? The fact that is very verbose? At first I wasn't a fan, but over time I've grown to really like it. That never happened to me with pandas, always felt the syntax was messy

▲mihaelm an hour ago | parent [-]

The verbosity takes a bit to get used too, but it sure beats the anything-goes feeling - messy as you put it - of pandas.

▲geysersam an hour ago | parent | prev | next [-]

I agree sql is more elegant. The problems arise when you have to add logic on top of sql. Often I end up constructing queries via string manipulation and that is not very ergonomic. Polars api is more verbose and complex than sql but at least it's not meta-programming.

The duckdb python api is okay, but it is a bit limited, no ctes, no as of join, and it can be slow at bind/interpretation time when you do stuff like unioning multiple relations in a loop (I think that becomes O(N^2), but I might be wrong). Most issues can be worked around, but Polars is designed from the ground up to be used from python.

▲gpugreg 2 hours ago | parent | prev | next [-]

You can query polars data frames with SQL: https://docs.pola.rs/api/python/stable/reference/expressions...

Unfortunately, polars does not support parameterized queries, so the risk of SQL injection is extremely high.

▲fzumstein 3 hours ago | parent | prev [-]

I tend to agree. SQL may have been harder to write in the past (worse autocomplete than pandas/polars), but now that AI is writing the code, SQL is usually much easier to read. So DuckDB is another interesting alternative to pandas.

▲refactor_master 3 hours ago | parent [-]

The cool thing about polars is that you can conditionally collect expressions over many layers of business logic, and then compute the result at the end. Doing this in SQL ends up in a hodgepodge of strings and trimmed ends to please the syntax. You can also pretty effortlessly write quite complex conditionals directly in polars, and bridge it easily to the surrounding python.

I find that SQL is only easier to read with minimal abstraction, but as soon as the project gets bigger SQL becomes an unwieldy island of different that has served its purpose after we’re done with reading/writing the data.

▲fzumstein 2 hours ago | parent [-]

This sounds interesting! Do you have a specific example by any chance or blog post/doc references?

▲refactor_master an hour ago | parent [-]

It’s just the lazy/expression part of the API, which is really the bread and butter of polars, rather than just being “replacement syntax” for pandas. This allows you to tap into abstraction that SQL can’t keep up with:

  import polars as pl

  # 1. Base Dataset
  lazy_df = pl.LazyFrame(
    {
      "store_id": ["S01", "S02", "S03", "S04", "S05"],
      "revenue": [5000.0, 2400.0, 15000.0, 900.0, 3200.0],
      "margin": [0.45, 0.30, 0.60, 0.15, 0.50],
      "tx_count": [120, 45, 300, 20, 85],
      "returns": [5, 12, 45, 2, 8],
    }
  )

  # 2. Define Layer Abstractions
  def get_kpi_layer() -> list[pl.Expr]:
    return [
      (pl.col("returns") / pl.col("tx_count")).alias("return_rate"),
      (pl.col("revenue") / pl.col("tx_count")).alias("avg_order_value"),
    ]

  def get_threshold_layer(thresholds: dict[str, list[float]]) -> list[pl.Expr]:
    return [
      (pl.col(col) > limit).alias(f"is_{col}above{int(limit)}")
      for col, limits in thresholds.items()
      for limit in limits
    ]

  def get_interaction_layer(numeric_cols: list[str]) -> list[pl.Expr]:
    return [
      (pl.col(a) / (pl.col(b) + 1e-5)).alias(f"ratio_{a}per{b}")
      for i, a in enumerate(numeric_cols)
      for b in numeric_cols[i + 1 :]
    ]

  def get_segmentation_layer() -> list[pl.Expr]:
    return [
      pl.when(pl.col("margin") > 0.4)
      .then(pl.literal("High"))
      .otherwise(pl.literal("Low"))
      .alias("margin_profile")
    ]

  # 3. Consolidate and Execute Single Graph Pass
  thresholds = {"revenue": [1000.0, 5000.0, 10000.0], "tx_count": [50, 100, 200]}
  numeric_cols = ["revenue", "margin", "tx_count", "returns"]

  expr_pool = [
    *get_kpi_layer(),
    *get_threshold_layer(thresholds),
    *get_interaction_layer(numeric_cols),
    *get_segmentation_layer(),
  ]

  final_df = lazy_df.with_columns(expr_pool).collect()
▲arn3n 2 hours ago | parent | prev | next [-]

The decision to default to the streaming engine is really interesting. My intuition is that this would be slower than other data frame operations that are more parallelizable with batch processing, because streaming engines necessarily process rows sequentially. Is my intuition off/am I overestimating how much auto-parallelization polars does?

▲orlp 2 hours ago | parent [-]

Streaming here has a different meaning than perhaps what you're used to. It's not referring to online processing where you maintain aggregates/state while an endless stream of data comes in.

The name was chosen early on to contrast with the old execution model, which was essentially all-data-in-memory, column-at-a-time. That engine still exists, we use it as a fallback mechanism for things that aren't supported yet in the new engine (or if you explicitly ask for `engine="in-memory"`).

The new execution model first constructs a computational graph of nodes which communicate in streams of in-cache batches (morsels) of data, meaning the full dataset will never be held in memory if not necessary. This was called the streaming engine for that reason in an early prototype and the name stuck. In hindsight I do admit the naming choice is somewhat confusing.

▲arn3n an hour ago | parent [-]

Cool, thanks for the explanation!

▲bluebarbet 2 hours ago | parent | prev | next [-]

What does this project have to do with Serbia? Are the developers in Belgrade?

▲mkl 38 minutes ago | parent [-]

It's just a play on the name, and it's pretty common. claude.ai has nothing to do with Anguilla, John Romero's rome.ro has nothing to do with Romania, twitch.tv has nothing to do with Tuvalu, etc.

▲bluebarbet 21 minutes ago | parent [-]

Indeed, and Bit.ly has nothing to do with Libya, nor Lemmy.ml with Mali (both failed states). I posit that domain hacking is an ugly, shortsighted, unserious habit that we should drop.

▲luciana1u 27 minutes ago | parent | prev | next [-]

every major version of polars is a reminder that the API you finally memorized was always just a suggestion

▲rfgplk 3 hours ago | parent | prev [-]

Seeing "release will land in the following weeks" kind of immediately turns me off.

▲thibaut_barrere 3 hours ago | parent | next [-]

I like when large projects do that. This gives leeway for sister projects (eg wrappers) to anticipate, room for apps that use it intensively to test things out (release candidate etc), something which has really helped me in the past.

In that specific case I use a Polars wrapper in Elixir (called Explorer) all week long, and I am very happy they are giving us early hints.

▲rfgplk 9 minutes ago | parent [-]

I was referring to the "land" verb choice, aka a clear Claudism. In fact looking at it more carefully, the whole post seems to be heavily AI written with minimal human intervention.

▲marliechiller 3 hours ago | parent | prev | next [-]

What is your understanding of a Pre-Release then?

▲dbdr 3 hours ago | parent | prev | next [-]

Why?

▲rfgplk 10 minutes ago | parent | next [-]

Clear Claudism. It wants to "land" everything, everywhere.

▲irpap 3 hours ago | parent | prev [-]

I assume because “land” is a word Claude would choose.

▲mgaunard 3 hours ago | parent | next [-]

Claude's wording (and knowledge) is based on what competent senior engineers would say.

▲blitzar 14 minutes ago | parent | next [-]

I really hope I don't have to work with these "competent senior engineers" anytime soon — my codebase couldn't handle it.

On the plus side I would spend all day hearing I am "absolutely right" from a superior being.

▲tecleandor 3 hours ago | parent | prev [-]

None of the competent senior engineers I know say "not this, but that" every two sentences.

▲pepperoni_pizza 29 minutes ago | parent [-]

But what about load-bearing seams that have provenance?

▲tancop 2 hours ago | parent | prev [-]

It's the only good word here. "Drop" can also mean the opposite and anything else sounds too formal. Don't get me started on "release will release".

▲rfgplk 6 minutes ago | parent | next [-]

"Polars 2.0 will be released (publicly) in the following weeks" sounds far more natural.

▲gjm11 27 minutes ago | parent | prev [-]

Well, they could have written something like "The 2.0 release itself will be in the next few weeks" or "It will be a few weeks before the actual 2.0 release".

(But I agree that "land" is fine here, and the rest of TFA doesn't strike me as obviously AI-written. And I'm not a fan of the "look, they did one thing that AIs often do! Must be AI and therefore bad!" thing in any case.)

▲NSPG911 an hour ago | parent | prev [-]

release will be [released] in the following weeks