> Those two combined means that unless LLMs reach the state of 'good enough'

Once you are able to not just go on-premise, but on-device, there a boatload of other advantages than efficiency (network independence, privacy, etc.) that make it attractive.

Tiny language models are already plenty "good enough" for a lot of on-device use-cases. E.g. there are a lot of things, that can be reformulated as ranking problems, where you can apply pairwise ranking, which even small language models are good at.