It seems like caching the relevant portion of PyPI in advance and turning the network off shouldn't be beyond what OpenAI can do?

I agree. I hope they learn that lesson and run future evals like this without pulling from external packaging repositories at all.

With my PSF board member hat on I'd like it if a whole bunch of other companies that make extensive automated use of PyPI (Microsoft and GitHub Actions for example) would do a better job of maintaining their own shared local caches.