On the other hand, this is helpful to know as a user of LLMs because it suggests that LLMs are bad at isolating the math problem from the cat fact. That means providing irrelevant context may be harmful to getting back a good answer in other domains as well.

Ideally you'd want the LLM to solve the math problem correctly and then comment on the cat fact or ask why it was included.