"every critique of AI assumes to some degree that contemporary implementations will not, or cannot, be improved upon."

They're token predictors. This is inherently a limited technology, which is optimized for making people feel good about interacting with it.

There may be future AI technologies which are not just token predictors, and will have different capabilities. Or maybe there won't be. But when we talk about AI these days, we're talking about a technology with a skill ceiling.