Is it really possible to get determinism while maintaining good GPU utilization? Parallelism alone introduces enough non determinism that I only see this really working on maybe a CPU?

Ish, you give up about 20%.

https://thinkingmachines.ai/blog/defeating-nondeterminism-in...