It reminds me conceptually of the idea of using a ST:TNG replicator to just give you another replicator of your own, or asking a stereotypical genie for "infinite wishes". The genie is indeed out of the bottle in many ways.

And for a lot of non-frontier purposes these days, you can bootstrap via LLM-as-judge so your hyperspecific wakeword model or whatever can be trained with little to no human input, that aspect of it is fully terrific.

The frontier models are a replicator that can give you another replicator which specifically produces tea, earl grey, hot, when you push the single button, and does nothing else.

I guess it’s more like asking the paid genie to give you a new cheaper genie.