The big difference you're not looking at is the memory bandwidth. The M5 Pro has nearly twice the one of the M6:

170GB/s memory bandwidth Configurable up to 32GB unified memory

307GB/s memory bandwidth Configurable up to 64GB unified memory

That is also the factor you want to take into account for AI inference, of course.

Both are still very low for AI btw.