> This was a clean-room implementation (Claude did not have internet access at any point during its development);

This is absolutely false and I wish the people doing these demonstrations were more honest.

It had access to GCC! Not only that, using GCC as an oracle was critical and had to be built in by hand.

Like the web browser project this shows how far you can get when you have a reference implementation, good benchmarks, and clear metrics. But that's not the real world for 99% of people, this is the easiest scenario for any ML setting.

> This is absolutely false and I wish the people doing these demonstrations were more honest.

That's because the "testing" was not done independently. So anything can be possibly be made to be misleading. Hence:

> Written by Nicholas Carlini, a researcher on our Safeguards team.