What do people think of property based testing? I’ve set up a few property based tests and it feels almost like formal verification.

I know it’s not the same thing, but PBT’s can catch all sorts of absolutely crazy bugs that I never would think to check for. Stateful stuff, cross-service stuff. My only gripe with PBTs has always been the tedious setup, but LLMs solve that.

I haven’t seen much discussion around PBTs lately, but you get a ton of bang for your buck in the LLM age

PBT testing and fuzzing are basically the same thing in my mind.

What's funny though is that the reasoning needed to say some sampling is comprehensive....makes it closer fo proofs! AFL — coverage-based fuzzing — was the original innovator here, but imagine an interpreter — code coverage in the interpreter itself isn't so good, you also want to explore the interpreted program!

This is why "is my testing comprehensive?" is actually an open-ended problem that, in the limit, is just as hard as formal verification.

I suspect we're basically going to see a convergence here. Randomized testing can be called "Monte Carlo formal verification" hehe.

Property based testing is great because it’s implemented as a library in many languages. You don’t have to rewrite your codebase in Lean.

Yeah, LLMs make setting up the annoying things like PBT and mutation testing easy. It’s great.