Yeah but you don't test for stuff you don't know you have to test for. All tests can pass and internally its still done in a braindead way that will explode at some point.

Absolutely, and this is exactly why humans are weaker than LLMs. We consider what is likely, they consider what the data/logic shows is possible. So you get edge cases tests/flagged, that wouldn't even be something a real person would know to look for.