IMO we're a step before that: We don't even have a real fish involved, we have a character that is fictionally a fish.

In LLM-discussions, obviously-fictional characters can be useful for this, like if someone builds a "Chat with Count Dracula" app. To truly believe that a typical "AI" is some entity that "wants to be helpful" is just as mistaken as believing the same architecture creates an entity that "feels the dark thirst for the blood of the living."

Or, in this case, that it really enjoys food-pellets.