Remix.run Logo
▲ bityard 16 hours ago

You don't mention what are you trying to show/investigate, though?

At first glance, this looks like another one of those "LLM riddles" that humans _think_ should be easy for an LLM to answer but is actually quite difficult because of how they work in the first place. The answers to such riddles ("should I walk to the carwash" or "how many R's in strawberry") reveal the weaknesses in our expectations of LLMs in general, not weaknesses or characteristics of any particular model.

I'm not sure I see this as much different than asking a bare model its own name: without a system prompt or post-training, it doesn't know, it's just a bag of weights and will hallucinate an answer to that the same way it will anything else.

I'm sure you already realize this but to be very explicit, You're not getting an actual random word out of an LLM this way. You're seeing the bias in each model's training set around how often they've seen "random word" followed by "lantern" or "zephyr" during training.