I guess public datasets on HuggingFace and some shadow libraries content is enough to start.

e.g. fineweb dataset is 50TB https://huggingface.co/datasets/HuggingFaceFW/fineweb