> before LLMs I definitely thought coding would be the last thing to go.

While LLMs do still struggle to produce high quality code as a function of prompt quality and available training data, many human software developers are surprised that LLMs (software) can generate quality software at all.

I wonder to what extent this surprise is because people tend to think very deeply when writing software and assume thinking and "reasoning" are what produce quality software. What if the experience of "thinking" and "reasoning" are epiphenomena of the physical statistical models present in the connections of our brains?

This is an unsolved and ancient philosophical problem (i.e. the problem of duality) of whether consciousness and free will affect the physical world. If we live in a materialist universe where matter and the laws of physics are unaffected by consciousness then "thinking", "reasoning", and "free will" are purely subjective. In such a view, subjective experience attends material changes in the world but does not affect the material world.

Software developers surprised by the capabilities of software (LLMs) to write software might not be so surprised if they understood consciousness as an epiphenomenon of materiality. Just as words do not cause diaphragms to compress lungs to move air past vocal cords and propagate air vibrations, perhaps the thoughts that attend action (including the production of words) are not the motive force of those actions.