Remix.run Logo
rozab 5 days ago

This study itself and also the media coverage of it are shockingly bad. I wrote a bunch about it at the time and I don't really want to do that again but here is the low down:

- This is not a longitudinal study. Each partipant did 4 20 minute sessions. It just happens that the total study took 4 months. - The paper does not imply long term harm of any kind, they just measured brain connectivity during the short tasks. - It is not surprising that when asked to use an LLM to write an essay, partipants don't remember it. They didn't write it. - It is not surprising they showed less brain activity. They were delegating the task to something else. They were asked to. - I think the authors of the paper deliberately attempted to obscure this. Q7 on p30 is "LLM group: If you copied from ChatGPT, was it copy/pasted, or did you edit it afterwards?" This has been removed from the results section entirely, and other parts of the results do not match the supposed methodology. - The whole paper is extremely sloppy, with grammar mistakes, inconsistencies, and nonsensical charts. Check out Figure 29...