Have you tried to make LLMs do that? Unless it's implementing previously developed algorithms they screw it up so repeatedly it's incredible. (And even when it is clearly defined it's obviously their weak spot).

The "rapid self improvement" is going to be ever further down the LLM rabbit hole where brute force, tenacity, and symbol juggling beat having a reasonable model of anything else. This is enormously more likely to recommend improvements to its own architecture than to develop a sudden grasp of molecular biology.

There isn't an AI bubble, but I'm increasingly convinced there is an LLM bubble, and that this noise is to do with preventing people noticing that before they've managed to pull something else out of the oven.