I would probably dig into the reasons for the differences in the benefit on the test machine and in prod

I had an issue like this for optimizing pgrust. I had an optimization that showed no impact on my test machine (c8g.4xl) and showed a 20% improvement when ran on my mac. It turns out the issue was the instruction cache on the c8g.4xl was being saturated on the test machine but not on my laptop, moving the bottleneck to a different place

If you can consistently reproduce the performance difference, you're already half way there

Fuzzers are brilliant at this and produce all kinds of insane floating point inputs.

On the bright side it could probably run for years without hitting the mistake as well. But it is nice to get it out of there.

> A question on 20s postgresql time - It does not look like you are accounting for reading data from disk

I choose the data size so that it would fit in memory on the machine I was testing on. fwiw, there's still a ton of overhead Postgres has that the toy example does not. For example Postgres will serialize the numbers into tuples and need to deserialize them to execute the query. That's why it's not an apples-to-apples comparison

You can try it. We're happy to help you with it, but expect there to be issues to work through. You would want to do it for something non-critical

pgrcolumnar is not the default storage method. Right now, it's exposed as a table access method. There's lots of design space for how to do this so I want to avoid pre-committing to anything

I want to build the best database possible. While Postgres is great, there are a lot of core issues that have been around for over a decade. We're working hard to get pgrust production-ready, and it will definitely be production-ready in the near future. I wouldn't be putting hundreds of thousands of dollars into this project if I didn't think we could build a production-ready database.

That's a pretty gloomy view

No, it doesn't. Different algorithms can vary pretty wildly in performance based on the design of the hardware they run on. For instance cache sizes can make one implementation of a sort on a certain sized dataset faster or slower than another. You can have a theoretically fast algorithm that just isn't as cache efficient as a theoretically slower one in big-O terms. All levels of the memory hierarchy as well as storage have specific bandwidths and latencies that inform the real world performance results. Parallelism is another issue. Many cores can do work very fast when you're careful about how you split up work between them, taking into consideration the synchronization latency and individual cache sizes and so on. The best approach for doing work on 64 cores can be dramatically different from what works best on 1 or even 2.

That is not a given. A database server can run faster on better hardware because it more effectively and aggressively caches things in memory, which can hurt it on lower-spec systems. Or it can better utilize SIMD instructions that are not present on the low end. Or it is more effective at utilizing more threads, but is slower when run at a low threadcount, etc etc etc.

If I'm not mistaken ClickHouse's min spec is quite steep.

There is nothing wrong with wanting to be compensated for your work, but for people like myself which use a cloud managed DB solution (GCP CloudSQL PostgreSQL) it means something like this would never be available.

I consider AGPL a poison pill in my work. That is not true with a suitable commercial license, although I expect a lot more commercial product (support/features/etc). As you note, your objective is to prevent commercialization of your software, but radically speeding up analytics is primarily a concern of large organizations so it seems like a mismatch in purpose.

> If AGPL is an issue for anyone, we would be happy to dual-license under a commercial license.

You should be setting up a CLA process then ASAP if you are looking to take outside contributions.

Sure, that's your prerogative, and kudos for not talking up open source. I'm not amazon size so can't use it, and AGPL is a no go for DB, don't want to be forced to open source my app because I use this!

Will await a MIT based fork myself.

If it is indeed 300* faster, I'm sure more rational corporations will rethink their license policy or be left in the dust.

> It’s weird because those who actually care about optimized pg gains are most likely large corporate customers. Why make a product targeting them and license it in such a way they’ll never use it?

Wait, sorry, you're asking why make something enterprise customers might pay for, and then not give it away to them for free?

AGPL seems like the right choice to me. I’m tired of companies like PlanetScale taking PostgreSQL, building a business on top of it, and then acting like PostgreSQL is theirs to control.

Most official Rust projects are dual MIT/Apache licensed by convention [1] (and most Rust libraries from third parties I've seen that are open source MIT follow the MIT/Apache dual license), so seems like this library shouldn't just be AGPL for a typical political choice of a Rustacean?

[1]: https://rust-lang.org/policies/licenses/

main indeed has two commits, but it clearly states the location of the rest of the commits, so I wouldn't be critical of main itself.

  hey claude, do a breakthrough
  You can find the actual git history at the v0.2 github tag.
  
  Co-Authored-By: Fable <noreply@anthropic.com>

It's easy to claim AI slop nowadays, but you should still mistrust-but-verify.

4K stargazers in a week.

We disabled parallelism in the blog post for demonstration purposes. The 300x slower refers to the clickbench numbers[0] where parallelism is enabled

[0] https://benchmark.clickhouse.com/#system=+liH|pgrs|gQ&type=-...