> by what mechanism is an LLM going to help you design and manufacture a novel bioweapon? We know they cannot do spatial reasoning, and if they were this good at bio stuff then drug research would be absolutely out of control.

I guess my question would be how good the LLM is at writing a program that is good at spatial reasoning? If it can create a valid test suite from existing art on the subject it might be able to bumble its way to something specialized that it can then direct at a higher level.

Isn't it that kind of 'rapid self-improvement' that is the concern expressed in the linked statement?

Have you tried to make LLMs do that? Unless it's implementing previously developed algorithms they screw it up so repeatedly it's incredible. (And even when it is clearly defined it's obviously their weak spot).

The "rapid self improvement" is going to be ever further down the LLM rabbit hole where brute force, tenacity, and symbol juggling beat having a reasonable model of anything else. This is enormously more likely to recommend improvements to its own architecture than to develop a sudden grasp of molecular biology.

There isn't an AI bubble, but I'm increasingly convinced there is an LLM bubble, and that this noise is to do with preventing people noticing that before they've managed to pull something else out of the oven.