Couldn't someone build an adaptive system, where the llm is frequently judging the difficulty of a task and switching to a more/less powerful model?

This only works well if you are creating sub agents with clean contexts for each task. If you constantly are switching models part way through some work then the whole session needs to be replayed each switch. You lose all the benefits of the context cache.

The judgment of task difficulty is a cursed problem. In most domains you really don't know what is down there until you start digging a little bit.

definitely something that will come sooner than later. In my view, adaption of model routing / model orchestration and mixing providers is just natural next step.

And while you're at it, solve the halting problem.