> it will be far less obvious what is AI-generated

Maybe not: https://mindmatters.ai/brief/ai-gen-writing-model-collapse-i...

After playing with googles new model for a day it is a breath of fresh air comparatively - but it has ticks a well.

Unlike code, or math where one can apply a quantifiable measure to what is being added in natural language is much harder to gauge.

I think there is a quantifiable measure-create a test dataset containing questions and human expert-written answers; give the questions to an AI to get corresponding AI-generated answers; then ask humans to guess which answer was human-written and which was AI-generated; now you can measure how much better than chance the humans perform at AI output detection, which gives you a quantifiable measure of the human-likeness of the AI responses.