Even with your explanation, this whole thing seems weird and scammy. What's the point of submitting something that is so incomplete and heavily tailored to this one benchmark outside of self promotion?

Benchmarks should be used for measuring general purpose query engines, not be an end in themselves.

This seems like a prime example of Goodhart's law. This benchmark is now just being used as a target for self promotion.

What's the explanation for paths to q00.rs-q36.rs being in the binary?

Thanks for the explanation, can you release the source of 0.4-preview so that people can understand what was going on?

thanks for your explanation. I'd love to see a concise explanation of the difference in benchmark performance between -preview and -beta

that would help clear the air, if the full-sql version has similar performance to the partial-sql version

thanks for the explanation, let's first see if the binary matches.

Honestly, if the database is not able to be run with a variety of queries other than the one used by ClickBench then don't submit it to the benchmark. Every other database in their list can run anything you throw at it. Hiding behind messaging that it was an early 0.4 build that can't do everything is ... weird! Just don't submit it to ClickBench if it is not usable by the world yet.

I hope they don't allow pgrust in again until the project matured. Otherwise the benchmark will be useless as it gets flooded by many projects that can run ClickBench fast but have questionable support for everything else.

> For the most part, the actual accusations are false. This submission was an earlier version of pgrust 0.4 called pgrust 0.4-preview.

This is a non-sequitur. It doesn't matter what the earlier version is called. It is what you submitted, and it is what the accusations are against. You don't get to create a new submission that (allegedly) fixes all of the issues and then say that the previous accusations are "false". Downplaying the issues by calling it a 'preview' when you submitted it yourself is also a bit of having your cake and eating it too, don't you think?

> ClickBench has no rules about how much SQL you need to support.

It is because of people like you that we have warnings not to eat tide pods, Jesus Christ.

Can you explain "PGRUST_GRAVITY_Q20_VARIANT"?

[flagged]

Should be outright banned as a precaution.

Two possible paths lead to their cheating code getting into a release. Either it was added by coding agent and no one looked at it, bad, but worse would be that a person was aware of it and approved - that's malicious.

Either way you don't want to be platforming the product because neither shows any degree of responsibility.

Hope the ClickHouse maintainers stand their ground and do not let pgrust back in until it and the maintainer has matured more.

Love the confidence and complete lack of acknowledgement of the initial fuck up.

Maintainer of pgrust trying to get it merged in is insane

written with claude lol

In the past, we've done that because we've tended to do large refactors, many of which we've backed out of. For that reason, we've kept our own internal development version, which is ahead of the public version. Then when whatever we're developing is ready for public use, then we open source it

That model of development is what led to Eric Raymond’s seminal The Cathedral and the Bazaar essay.

“Cathedral” refers to development behind closed doors, where you just get a release and a source dump to go with it.

“Bazaar” refers to development done in the open, where you have constant access to the bleeding edge.

At the time of the essay, GCC and Emacs were ESR’s examples of Cathedral-style development, and Linux was the premiere example of the Bazaar style.

The Cathedral style has become a lot less common over the years, but it’s still how e.g. SQLite operates, I think? E.g. Google definitely operates like that with AOSP, but that's arguably more a fruit of them being maliciously compliant with their Open Source promises.

Not normal. This a new trend of people who use too much Ai, or maybe more specifically the Claw/Ralph loops where you end up with a few commits touching 1000s of files. For about the last year, many Show HN to github projects are often just a single commit

We need to think more about what we are doing as an industry, instead of letting the machines think for us (imo)

You thought the vibe coded postgres port was a serious project?

first time I found out about it was on reddit

https://www.reddit.com/r/rust/s/QJ5uNP5XMT

The LLMs are, they wrote them after all.

they are "For the record"

Good choice. I'm sure version 0.5 will be the version where this vibe coded rust rewrite will finally be free of drama once and for all.

> The link goes to a removal request

Incorrect, the link is to a PR from the creator of Clickbench and already merged. It is not a "request".

> alleged cheating

It was more discovered than alleged, but perhaps "cheating" is an inaccurate word as well, this is just Ai being Ai and a human obviously doing eyes free coding.

confirmed cheating, not alleged

To be fair Github refuses to render diffs after a couple of hundred lines.

Yup. These are people who have gone full Claude. The dopamine seeking is at previously unseen levels. they think this pr is some factual item that means something so they post it on HN as click bait. But no it only means something to a fulltime ai slopper

It was certainly assisted by AI, but there isn't an agent out there that does that kind of cheating without deliberate prompting by the operator.

I've taken a vibe-coded project pretty far, with even better testing than most other things: the main factor is that it's big on falsifiability, i.e. it has to see a test for a new feature actually fail before it considers the test any good. Not just the red/green from being originally unimplemented, it does a mutation test to ensure it flips back to red. But wow did it ever make that test framework complicated, byzantine, and ultimately brittle until I stepped in and forced a consolidation and simplification pass (twice now). Part of the complexity came from it being a polyglot monorepo, but I've found even in other places, you have to follow an expand-contract cycle of development when vibe-coding. Not all that different from any other system you've built up organically, just on a much faster cycle.

But back to my original point, none of this really applies to the pgrust benchmark, which was obvious and blatant cheating on the human's part.

We were the agents for the models all along, with our highly adaptive tools: versatile sausagey appendages and flapping fleshy orifices for interacting with the world

Would the AI have bothered to maximize the benchmark, if it hadn't been asked to?

"LLM" isn't the problem. Asking an LLM to explain cheating is a good goal. Asking it to "Maximize the benchmarks" reeks.

[dead]

I suspect the author of that PR submitted it and attempted to correct the mistake because he got caught. Forgiveness because he reneged after getting caught encourages the behavior. The project, nor author should not be trusted; otherwise is naive.

> acknowledges the issue

I don't see any acknowledgement in that text.