If we are about to achieve the AI rapture and models are solving Millennium Prize Problems, where is my AGI??? I just tried for weeks to write a simple paper with Claude and even though I put a ton of effort in context engineering, material collection and prompting it degraded quickly into an unintelligible mess of a word salad. Mind you that was with Fable 5.1, a model that supposedly is so sharp that it’s under export ban in the US for certain domains. Kind of hard to fathom seeing it fail so hard on a simple task that I was then able to do manually in the end.

So let’s stop pretending these things are going to replace any serious knowledge work in the coming years, I don’t think they can make them that much smarter for such tasks as I can’t see any difference between the latest model updates, make them write anything more than a few pages of text in a novel domain and it sounds like a stochastic parrot. And I have tried so many strategies to make it reflect on what it writes, it just doesn’t have the ability to think like a human can, it just simulates thinking to a degree that is convincing while you only look at a small sample of its output. I found it’s the same with code, small changes work reasonably well and you can’t see the defects but keep doing that for a few months and you’ll have stochastic code soup. Not sure if these things make me even 2x more efficient in the long run for producing new stuff. They make me faster for search and lookup, but not anything that requires thinking beyond the simplest mechanical form. And it doesn’t help me that it can solve a Millennium problem, which btw it can only do while being guided by a team of world class mathematicians. So it’s at best an ability multiplier that’s tethered to the human intelligence steering it.

This is actually a good point.

Where are the weird discoveries? Something like "super regular expressions", some patterns that are just as simple but way more expressive. Where is a better axiomatization of arithmetic than Peano's? Can you make a Y combinator out of B and M primitives?