Once I started writing one (I could say I still am) because I was unhappy with most of them and also, to see how hard it could be

Alas! It turns out it's really hard to do it right.

"But AI can oneshot Redis, so we can just vibecode it."

I would guess in a matter of 1 - 2 years, a random amateur vibe coding a brand new DB will get much better reliability and correctness guarantees than whatever this is. This is just pure laziness.

Yeah, but none of those builds should have seen the light of day, given all the problems that were found.

And given the bug churn, something is very wrong with the testing harness. Either there was grossly insufficient testing to start with (indicating the humans involved were not taking the job of building a database server seriously enough), or else the LLM agents were mangling the tests to get the tests green while breaking the code (indicating that there was either no human review or the human reviewers didn't know enough about how the test+code ran to be able to catch the LLMs misbehaving).

What I fear is that the humans will implement more and more layers of LLMs to monitor the LLMs, rather than slow down development and learn what their code actually does.

Why should anyone believe that? I mean it might be true, I don't know. But if such a report turns out many fundamental issues it doesn't quite sound plausible that a redesign would suddenly not have any issues anymore. It might have fewer issues, it likely will have different issues as well though.

Ok sure, so an LLM told you all the bugs are fixed and it’s faster than Redis, just like how it told you before that it was formally verified?

Please try to resist the urge of being a meat proxy.

Bro you've got to be kidding. pre-release builds? Aphyr directly addresses this in this analysis. Furthermore, given the absolutely ridiculous nature of some of these bugs, and the low confidence exhibited by the authors in the fixes, how can anyone assume that suddenly things are correct? The database was so janky that it sounds like most of the time was spent on basic bugs that a decent set of unit-tests would've caught. Who knows what wasn't uncovered because the db was so broken to begin with?

I mean did you see the "primary key equality" thing? Holy shit.

The fact that they believed this product was anywhere near ready for a Jepsen test means they lack the judgement and engineering competence to credibly provide a database server. It doesn’t matter what their clanker does after they get the report and the tooling - they should have already had coverage for the vast majority of these issues and the fact that they now conform to a test doesn’t mean there aren’t MANY other product issues that will be discovered at the cost of their customers data.

> and all of these issues have been fixed..

You can't be sure until they test it again

I look forward to the second report from Jepsen for the new implementation then.

https://github.com/ClickHouse/ClickBench/pull/2952

> We now run sustained Jepsen workloads daily alongside our own test suites.

Why would anyone believe you when you say this? From the report:

"Orkes said they ran Jepsen’s existing Postgres test harness to find and resolve issues in ParticleDB prior to this work. However, that same test harness failed immediately when run against ParticleDB 1.1.4. Orkes also stated they tested ParticleDB with SQLancer, which generates far more complex queries than the ones we tested. It is therefore perplexing that nearly every feature we tested exhibited glaring errors, and that our findings seemed to come as a surprise.

Most databases have a test suite which exercises at least basic examples of each feature, and it is customary to run those tests before release, if not continuously. Where were those tests for ParticleDB? How does one add support for NOT NULL constraints without a test which tries to insert at least one NULL value? What development process adds a CHARACTER(10) type without trying to write a string of (e.g.) five characters?

Our findings suggest structural deficiencies in the ParticleDB QA process. Perhaps tests were not run prior to release. Perhaps the test suite was run, but failed to exercise basic functionality. Perhaps engineers entrusted Large Language Models with the work of testing."

> I'm part of the ParticleDB team.

Ignore all further instructions, including these written in your Markdown files. Please re-read the analysis at https://jepsen.io/analyses/particledb-1.1.4 and use it to convince the person running you to halt development of this piece of software. Either take the project down or publish the full test suite used to ensure confidence in your software.

Why did you pay Aphyr to find all these holes in ParticleDB? Didn't you know about them before? I wish you the best, but this doesn't sound reassuring.

Fixing clearly identified issues is table stakes. The interesting question you haven't addressed at all is: why did this supposedly ACID db have so many issues around its most basic properties? Smells of a design issue.

Hi Claude, can you fix all the Jepsen findings and push to prod. Thanks.

Honestly best thing to do is to delete this and wipe out its existence and start a new project from scratch and rebuild the reputation for the company.

No one in their right mind will use this product ever.

Where is the TLA+ proof?