Does it matter? Hypothetically if these pre-training datasets disappeared, you can distill from the smartest current model, or have them write textbooks.

If LLMs happened 15 years ago, I guess that we wouldn’t have had the JS framework churn we had.