The point is that AI models do these kinds of things all the time. They're not really all that smart or intelligent, they just replicate patterns or boilerplate and then iterate until it sort of appears to work properly.

> appears to work

That "appears" is doing a lot of heavy lifting.

The code working isn't what's being selected for.

The code looking convincing IS what is being selected for.

That distinction is massive.