To me it’s morally ambiguous… if you hand parts of your thinking over to a tool like this (knowing full well the terms of service), of course the tool makers will want to claim some credit, and they do deserve it. But the bigger question to me is the scientific one: did their new model arrive at this result because it had closely-related training data from a human, or did it extrapolate to this line of thought on its own? The answer says a lot about how valid their claims of “AGI” are vs. a very fortuitously cherry-picked example.

It would actually be a really interesting study, if they would ever be willing to be transparent about this, how the result differs with and without his conversations in the training set. How quickly it arrives at the result, whether it takes the same approach, etc.

> Are they supposed to manually review all their data t

Yes. They should determine if training data included this teams data. Consider the money they spent, the press release and the purpose of their publication.

Since they failed to answer this question they shouldn't have published.

> What are you expecting OpenAI to do exactly if these mathematicians voluntarily submitted their prompts into ChatGPT's training data?

Personally, I would expect them to have a little class, to KYC, and to manually turn off training for known competitors using their service so as to avoid any unforced goofs like this.