This is amazing. With time, we should be able to write GPU programs semantically identical to user land programs.

The implications of this for inference is going to be huge.

CUDA programming consists of making a kernel parameterized by its thread id which is used to slightly alter its behavior while it executes on many hundreds of GPU cores; it's very different than general purpose programming. Memory and branching behave differently there. I'd say at best, it will be like traditional programs and libraries with multiple incompatible event loops.