In college I used mathematica a lot while taking linear algebra. I ended up having to relearn a lot of that math later since I never really understood it at a deep level, although I was able to use it and apply it under class conditions. I feel like a lot of my LLM derived knowledge is similarly superficial.

That's absolutely right. Math is difficult to understand without writing it down on paper. As one of my professors said, "something magical happens when you write down a problem from scratch and work it out". LLMs make people believe they are improving their productivity and that may be so, but it also removes that magic and removes that deep learning.