Yeah I agree, it's not about where it's at now, but whether where we are now leads to something with general intelligence and self improvement ability. I don't quite see that happening with the curve it's on, but again what the heck do I know.

What do you mean about the curve not leading to general intelligence? Even if transformer architectures by themselves don’t get there, there are multifarious other techniques, including hybrids.

As long as (1) there are incentives for controlling ever increasing intelligence; (2) the laws of physics don’t block us; and (3) enough people/orgs have the motivation and means, some people/orgs are going to press forward. This just becomes a matter of time and probability. In general, I do not bet against human ingenuity, but I often bet against human wisdom.

In my view, along with many others, it would be smarter for the whole world to slow down AI capabilities advancement until we could have very high certainty that doing so is worth the risk.