Deterministic seed
Deterministic seeds barely work on a single machine, small scale training run.
They just don't work at all on a many month long, 100K+ GPU cluster training run.