audio/video processing, scientific/technical/engineering computing, etc. may have wildly different performances when compiled for different x86-64 ISA versions

This is pretty vague and makes it sounds like there are big differences in instruction sets.

In actuality it comes down to memory access first which has nothing to with instructions.

After that it comes down to simple SIMD/AVX instructions and not some exotic entirely different instruction set.