> The fun thing is that even the strawberry issue tells us more about the limitations of llms than not. In other words, that error is useful

That's a fair way to look at it - failure modes tell us something useful about the underlying system. In this case it tells us something about how LLM's work at the token level.

But if you go a step beyond that, you would realise that this problem is solved at a _general_ level with the reasoning models. GPT o1 was internally named strawberry as far as I remember. This would be a nice discussion to have but instead of shallow dismissal of AI as a technology with a failure mode that has been pretty much solved.

What really has not been solved is long context and continual learning (and world model stuff but I don't find that interesting).

<< What really has not been solved is long context and continual learning (and world model stuff but I don't find that interesting).

I wonder about that. In a sense, the solution seems simple.. allow more context. One of the issues, based on progression of chatgpt models, was that too much context allowed for a much easier jailbreak and the fear most corporates have over that make me question the service. Don't get me wrong, I am not one of those people missing 4o for telling me "I love you". I do miss it its nerfed capability to go across all conversations. Working context is was made more narrow now. For a paid sub, that kind of limitation is annoying.

My point is, I know there are some interesting trade-offs to be made ( mostly because I am navigating those on local inference machine ), but with all those data centers one would think, providers have enough power to solve that.. if they so chose.