I mentioned the minimal proof assistant I elicited last night. When Fable 5.1 thought it was done, I asked how we knew the prover was sound (which means, in the jargon, that the theorems that it proves are actually true). It checked and immediately found five different ways it could "prove" false theorems with it (and fixed them).

Then it suggested writing a fuzzer to try to flush out more soundness problems. It loves fuzzers! And usually they are an excellent cost/benefit tradeoff. However, in this case, I questioned whether a fuzzer would ever actually succeed at finding proofs of a theorem it was set to prove, even unsound proofs, and after doing some tests it admitted that the fuzzer it had proposed would have been completely useless.

Then the conversation was incorrectly flagged as me working on a malicious security attack, so I was downgraded to Opus 4.8. I switched back to Fable, renamed the file in its scratchpad, and asked it to please use the term "generative testing" instead of "fuzzing". Thanks to not using the computer-security name for generative testing, there were no further false flags.

Then, this morning, I was reading the spec it proved its program fulfilled, and asked whether a certain trivially incorrect alternative program would also fulfill the spec (a misformalization problem rather than a soundness problem). It churned away for a while, discovering that while, actually, no, that program would be rejected, a different trivially incorrect program would pass, and credited me in the docs with pointing out the issue. I pointed out that in fact the issue it had found was completely different from my stupid misreading of the spec. It fixed the doc.

Fable 5.1 isn't Mythos but it's generally considered to be a "frontier model".

So, from my point of view, the whole experience has been a constant fractal of hallucination, in which I have to constantly struggle to keep my grip (and Claude's grip) on actual reality, because it's so willing to make up surface-plausible nonsense.

______

P.S. Also, in another task today, it thought pip wasn't installed and was trying to figure out how to work around it. But that's not so much a hallucination as a failure to recheck assumptions — I hadn't installed pip on the machine before the first Claude work on it, and it just assumed that was still the case. Also I think that might have been Opus rather than Fable, so it's not as strong a case.

I feel like this pattern is a great way to practice and sharpen critical thinking. It's like a whole new skill to deal with computing systems in this way.