"Irrelevant" facts about cats are the most interesting part of a math problem, because they don't belong there. The math problem was also "irrelevant" to the information about cats, but at least its purpose was obvious because it was shaped like a math problem (except for the interesting barnacle attached to its rear.)

Any person encountering any of these questions worded this way on a test would find the psychology of the questioner more interesting and relevant to their own lives than the math problem. If I'm in high school and my teacher does this, I'm going to spend the rest of the test wondering what's wrong with them, and it's going to cause me to get more answers wrong than I normally would.

Finding that cats are the worst, and the method by which they did it is indeed fascinating (https://news.ycombinator.com/item?id=44726249), and seems very similar to an earlier story posted here that found out how the usernames of the /counting/ subreddit (I think that's what it was called) broke some LLMs.

edit: the more I think about this, the more I'm sure that if asked a short simple math problem with an irrelevant cat fact tagged onto it that the math problem would simply drop from my memory and I'd start asking about why there was a cat fact in the question. I'd probably have to ask for it to be repeated. If the cat fact were math-problem question-ending shaped, I'd be sure I heard the question incorrectly and had missed an earlier cat reference.

On the other hand, this is helpful to know as a user of LLMs because it suggests that LLMs are bad at isolating the math problem from the cat fact. That means providing irrelevant context may be harmful to getting back a good answer in other domains as well.

Ideally you'd want the LLM to solve the math problem correctly and then comment on the cat fact or ask why it was included.

Exactly. The article is kind of sneaking in the claim that the LLM ought to be ignoring the "irrelevant" facts about cats even though it is explicitly labelled as interesting.