Writing needs a conceptual split analogous to the split between math and calculating.

Just as a calculating can be implemented on a computer which has low cognitive abilities but high algorithmic and procedural abilities, we need to extract out the word-smithing capabilities from writing separate from the thinking portion. Our lack of distinction in terms reflects a muddled conceptual framework.

LLMs are excellent wordsmiths completely divorced from the concept of thinking. They break the correlative assumption - that excellent writing is corresponds with excellent thinking. Until now, we've been able to discern poor idea because they have a certain aesthetic, think conspiracy rants in docx saying something about a theory of everything based on vibrations. But that no longer holds. We have decent enough word-smithing coupled with a deficit of thinking. Unfortunately this breaks our heuristics with consequences ranging from polluting our online commons to folks end up believing nonsense like ChatGPT named itself Nova and they are a torchbearer for spiritual gobbledygook.

My point is that we're in the process of untangling these two and as a result, we're likely to see confusion and maybe even persistent misunderstanding until this distinction becomes a more common part of how we talk about and evaluate written work. They're living in an AGI-world and we're just..not.