Orthogonal to the blog, but fun fact: Determinism is crucial for large-scale training runs, and effective RL-runs as it avoids collapse of numerics, but it costs more(about 20% perf) because you need can't parallelize over the batch nor use some specific instructions that dont guarantee reduction order in the way way. Horace He from Thinky has a great blog about this:

https://thinkingmachines.ai/blog/defeating-nondeterminism-in...