What they did is they gave the AI a huge pile of docs and said "implement this". But they didn't give the AI any tests as part of its objective. I was under the impression it's generally common practice to use a test-driven development approach with AI, so I think it would have been more representative for them to have split their test corpus in half: one half it could see during implementation, and then the could test on the other half.

Also, they restricted access from the internet, so I think this would have disadvantaged less popular languages.