Good question, and maybe I should have provided an alternative description rather than just criticising.

Unfortunately, it is actually surprisingly hard to pin down, and I think mathematicians (and, as a student, I count myself as one to some degree at least) now have the task of making this a lot clearer. If we want to justify our existence in the face of new machines that can seemingly ‘do our work for us’ (so far in a restricted context), we should give a robust defence of our practice. If we can’t do this, we simply don’t deserve the funding (which, by the way, again contrary to some misguided statements here, isn’t very much anyway!). I think all of this will become clearer to outsiders as time passes, but for now it’s not easy to give a quick answer — though I can try.

Mathematics is about understanding things. Isn’t that what every subject is about? Well, I suppose so, but mathematics more specifically does something like the following:

(1) observe some phenomenon in ‘reality’.

(2) attempt to formalise that phenomenon in such a way that it can be manipulated purely symbolically.

(3) use this (perhaps fairly arbitrary; remember that we can invent as many formal systems as we like) system to deduce from our initial assumptions new facts that would otherwise have been very non-obvious.

It seems like outsiders have a decent grasp of (3) and the application of AI to it, but have very little idea about the other two steps. It seems to be widely assumed among non-mathematicians that problems are essentially god given and that the job of a mathematician is therefore to chug away on these problems, manipulating symbols and trying out tools, in the hope of learning a yes/no answer to each one.

The first two steps are by far the hardest and most important, and they’re also the parts that AI seems currently unable to help with.

NOTE: this is not a deeply insightful description of what the subject is about, and there are many better characterisations out there. I think Tao and various others have written recently about why complicated and inscrutable AI-generated proofs aren’t nearly as valuable as one might imagine. (That’s not to say there’s no value to such proofs; perhaps in time, as technology improves, mathematicians will come to accept AI as part of the process.)

If you want to understand all of this issues better, reading the recent slew of guest posts on Tao’s blog would be a very good start.

Despite loving Mathematics and having considered being a Mathematician myself, that wasn't a very robust defense. Also, even though I'm a big Terence Tao fan, I'm not sure he has sorted that out this defense in full himself.

This post which he forwarded was quite poor in my opinion. Confusing, all over the place with AI criticisms and promotion of the AI hazing being done by mathematicians.

X thousand mathematicians who want to protect their livelihoods signed a bunch of letters against AI. Duh. We've seen similar movements from every profession that has been displaced ever.

Terence Tao uses AI and has made a few good points on how to use it. But defensiveness leaks into almost every defense of the role of humans in Mathematics that I've read, even his own at times.

To be clear, I actually believe that Mathematicians aren't going away, but I dont have enough knowledge about the life of a professional mathematician to articulate a path forward.

This "path forward" is what I'd like to see. We need a top mathematician with enough intellectual honesty (Terence Tao qualifies, I think) to start this questioning with "there's actually no role for human Mathematicians" as one of the options on the table and go from there.