I thought the big difference between the GPU and ANE was that you couldn't use the ANE to train. Does the GPU actually perform faster during inference as well? Is that because the ANE are designed more for efficiency or is there another bigger reason?

GPUs are usually faster for inference simply because they have more ALUs/FPUs but they are also less efficient.