They want to be power efficient, while Nvidia doesn't. That means a much higher cost (larger chip area) for the same performance.

I don't buy this as an explanation for the gap in the actual silicon's performance.

Nvidia's chips lose very little performance when they're severely power limited. In fact, that's why Nvidia's Professional versions of their GPUs are typically running at under two-thirds of their consumer equivalent's power draw.

Nvidia's cards are primarily designed for their professional applications where the power consumption is lower, and then they just juice up the consumer cards deep into the territory of diminishing returns just so they win some benchmarks.

On a performance-per-watt basis, Apple is still behind an Nvidia card with the juiced up power consumption, and when you dial back the power draw, Nvidia cards are miles ahead of Apple's silicon for performance-per-watt.

A GPU can be used for inference, but, for that use, there are much better choices. Apple designed their NPUs for that, IBM added an NPU to their mainframe chip and AMD and Intel are planning on adding inference-specific instructions to the amd64 ISA.