While I agree with you in the main, I also take seriously the "until someone can explain why" counterpoint.

Though I agree with you that your calculator doesn't understand math, one might reasonably ask, "why should we care?" And yeah, if it's just a calculator, maybe we don't care. A calculator is useful to us irrespective of understanding.

If we're to persuade anyone (if we are indeed right), we'll need to articulate a case for why understanding matters, with respect to AI. I think everyone gets this on an instinctual level- it wasn't long ago that LLMs suggested we add rocks to our salads to make them more crunchy. As long as these problems can be overcome by throwing more data and compute at them, people will remain incurious about the Understanding Problem. We need to make a rigorous case, probably with a good working alternative, and I haven't seen much action here.

  > "why should we care?"

I don't think anyone is saying that the calculator isn't a useful tool. But certainly we should push back when people are claiming it understands math and can replace all mathematicians.

  > If we're to persuade anyone, we'll need to articulate a case for why understanding matters

I'll say that a major motivating reason of why I went into physics in the first place is because I found that a deep understanding was a far more efficient way of learning how to do things. I started as an engineer and even went into engineering after my degree. Physics made me a better engineer, and I think a better engineer than had I stayed in engineering. Understanding gave me the ability to not just take building blocks and put them together, but to innovate. Being able to see things at a deeper level allowed me to come to solutions I otherwise could not have. Using math to describe things allowed me to iterate faster (just like how we use simulations). Understanding what the math meant allowed me to solve the problems where the equations no longer applied. It allowed me to know where the equations no longer applied. It told me how to find and derive new ones.

I often found that engineers took an approach of physical testing first, because "the math only gets you so far." But that was just a misunderstanding of how far their math took them. It could do more, just they hadn't been taught that. So maybe I had to take a few days working things out on pen and paper, but that was a cheaper and more robust solution than using the same time to test and iterate.

Understanding is a superpower. Problems can be solved without understanding. A mechanic can fix an engine without knowing how it works. But they will certainly be able to fix more problems if they do. The reason to understand is because we want things to work. The problem is, the world isn't so simple that every problem is the same or very similar to another. A calculator is a great tool. It'll solve calculations all day. Much faster than me, with higher accuracy, but it'll never come up with an equation on its own. That isn't to call it useless, but I need to know this if I want to get things done. The more I understand what my calculator can and can't do, the better I can use that tool.

Understanding things, and the pursuit to understand more is what has brought humans to where they are today. I do not understand why this is even such a point of contention. Maybe the pursuit of physics didn't build a computer, but it is without a doubt what laid the foundation. We never could have done this had we not thought to understand lightning. We would have never been able to tame it like we have. Understanding allows us to experiment with what we cannot touch. It does not mean a complete understanding nor does it mean perfection, but it is more than just knowledge.