Sure, but behaviors that never have a bug or regression don't get a test. Software of this kind of complexity has all kinds of behavior that has never been broken, and doesn't have a specific test written for it.

Getting an extensive test suite passing is certainly orders of magnitude better than having no test suite at all, but it still doesn't tell you as much as you need to know. I would absolutely never trust an LLM Postgres rewrite (in any language) in production based on "only" Postgres's test suite passing.

The maintainers that wrote those tests will have experience you won't get out of a rewrite.

I think this is also where the real work is. A rewrite is one thing, that you can show off with a flashy blogpost. The maintenance, for years to come, won't be of that nature yet it still requires as much work.

This feels like the image of the plane that returns from battle with bullet holes, and the engineer being asked to path up where the holes to make it stronger. Only to be told to patch where there weren't holes as those planes didn't make it home.

While not an exact fit of an analogy, those tests patch what was a problem with Postgres in the wild. What it doesn't cover are the things that worked in Postgres without tests, but may fail in port and go undetected.

One issue is those are the bugs you get when you write it in C++.

They aren't the bugs you get when you write it in Rust.

The kind of bugs you get are usually a function of the problem, language, implementation approach.

> Most extensive test suites are exactly production scars: every time you have a bug or a regression, you write a test that confirms correct behaviour.

If you can be 100% guaranteed that there indeed is a test for every occurred bug. Sometimes maintainers are not so strict about it.

And some programmers are so good that some issues are self-explanatory and they write good code to note a thing but don't write a test, because implementing the test is more expensive.

a code written to pass a test can surface unintended new bugs.

very naive. the runtime behavior of a rewrite should be significantly different in all kinds of unpredictable ways nobody see coming or might expect. It is a combination of language semantics, compiler behavior, operating system behavior, file system behavior, driver behavior, ..

A lot of the signal (github, forums, mailing lists, discord, etc.) can be turned into signal. Right now it's easy enough to collect. In future it will be easy enough to cluster and generate preferences, experience, etc.

Every bug report, code change as a result, PR / commit message, PR comment that steers preferences, etc. is solid signal to generate future tests.

> It’d be hard to pass its test suite and not be as robust as the original.

This is not true, even in principle, even for Postgres itself. You'd be right to say that it'd be hard to pass the test suite and not be robust at all to some extent. But even in Postgres, I bet that you can quite easily introduce a change that will pass the whole test suite but reduce robustness compared to the latest release (for a somewhat silly example, add a call to `exit()` on a timer that's longer than the longest duration test in the suite - that will significantly reduce robustness while still passing the entire test suite).

Sure but these scars/tests are from the original implementation. Just because it doesn't have issues there doesn't mean it didn't bring its own set of issues

This is all well and good in theory, but the number of times I've seen tests that don't actually test what they say they're testing is hard to count. Yes even when you encourage the developers to ensure the test fails first and do TDD. Tests help you ship with confidence but there's usually at least a few that are just passing by pure luck.

So no, I wouldn't judge a rewrite as being equal just because it passes the tests. That said, I don't think that means you shouldn't do it. You just have to be pragmatic about it.

Passing a regression test suite only proves that those particular regressions aren't present. It proves nothing about robustness beyond that.

They ought to, but are they? In https://wiki.postgresql.org/wiki/Developer_FAQ I don't see a requirement to provide a regression test for a bug fix.

You immply that a testcase exists for every weird edge case. Especially filesystem and concurrency is things you can barely build test cases for.

Even a 100% test coversge is far away from verifying all behaviour.

Edsger W. Dijkstra:

"Program testing can be used to show the presence of bugs, but never to show their absence!"