Really interesting.

On the other hand, this only applies to a limited set of workloads (training is often memory bandwidth bound for example).

I also wonder how this affects power consumption.

Training is compute bound.

If its memory bandwidth bound you need to raise your batch sizes to leverage more parallelism.

It's also useful for prefill in inference, VLLM sees up to 30% boost depending on the exact kernels and prefill size