I think you can train on math /science using synthetic generation to a significant extent. Training for coding requires more of the "scraping github / stackoverflow" angle but IMO that's a shallower hill to climb than scraping copyrighted art and literature.

There are actual models trained on ethical datasets but they are obviously not very high powered. If companies with the resources of an anthropic or openai were doing it (ha) it would be more feasible