They lack the empathy to understand where and why you're struggling.

I've given private math lessons and seen students struggle with ai, even though ai gave the right answers.

My intuition is that humans spot xy problems easier when teaching (user ask x but really needs y), whereas llms will oblige writing about x.

Ok, that's fair, and I think that AI isn't that useful for people who don't really know what they want to learn or do.

What I had in mind (and found personally useful myself) was being able to ask a chatbot to break something down into simpler and simpler concepts, until my weak fundamentals or lack of formal education could grasp some bit of it.

I'm not sure many humans (or colleagues at a workplace) would have the patience to tolerate the stupid questions I've been asking.

That's an interesting observation. I wonder if simply prompting for that in your personal system prompt might help?

"Be aware of 'xy problems': when the user asks x but really, they need the answer to y to further their understanding. ..."

The models are so capable now I think a lot of these real limitations can be solved by prompting, harnesses or fine-tuning.

That's interesting, thanks for sharing. Can you please share a specific example of what you describe? Does the student stop asking why after a while?