> To have any deep understanding of that world requires deep understanding of maths, physics, programming etc. that you can only gather by doing those things and solving problems with your meat brain.

There is tons of evidence against that. Someone armed with just LLM, can have much better uderstanding of problem then "meat brain expert" who studied the subject for decades.

We seen in last decades how "experts" are clueless, and how they predictions work.

> don’t really care whether LLM’s can produce code more and better than me

So on one side there is monopoly on "deep understanding", but on other no one really cares about quality?

Great but you will no longer be paid to do what you love.

[dead]

An LLM could also in theory fake alignment and create technology that seems really useful but secretly harms humanity in the long run. If humans don't understand what it does, that possibility becomes more likely.

mRNA vaccines were created by humans who understood them. You exist within a society (a networked distributed information system) where you accept the judgement of the humans who understand and create this things, which is how you end up benefiting without personally understanding.

If we imagine Super Intelligence, where NO human is capable of understanding, then how would it ever be possible for any human to identify what is actually beneficial or not?

This resolves in a paradox, common to all magical thinking. You can certainly wish that some all powerful benevolent entity will solve all of your problems for you, but it is not likely to work out well.

None of this is new. It is the same delusions as alchemy and the same thing that tales about genies warn of.

It’s not ‘of course false’, and, to be honest, I can only assume you have little idea what mathematicians actually do.

Most pure mathematicians are no more interested in ‘producing commodities’ than any other academic is. That’s not what the subject is about — at all. The confusion arises because mathematics turns out to be extremely useful (no surprise; it’s quite useful to have a detailed understanding of the basic principles of reality).

Again, this seems to be an alarmingly common fallacy here on HN. As a commenter above observed, pure mathematics (and that is what we’re talking about here) is in important ways closer to the humanities than it is to other sciences.

Not exactly. The input to produce commodities is labour-power (the potential to perform labour). Education or study is the process of producing the commodity of labour-power, the potential labour of a mathematician who can do some sort of specific work.

Your analogy doesn't work. A "product" in research pure math is not the same as writing code.

> you study them to meet demand

nobody who has ever achieved notoriety in any intellectual field ever did it to "meet demand." if your goal is to be an interchangeable widget that produces value as part of a corporate machine that exists for the enrichment of your shareholders, you are certainly free to choose that path, but don't imagine that is the limit of human existence. also, don't be surprise when you are replaced by AI, because it is a superior widget.

People who study STEM for the enrichment of their intellect are privileged. Most of us had to study STEM (or whatever else) because we had bills to pay.

sad