If matrix multiplication isn't the main primitive, I think we have a lot of pain coming our way.

Maybe that is far fetched, but I could see them specializing for some super high dimension multiplication and meanwhile 5 years later turns out "all you need" are 3x3 matrices and suddenly 90% of your specialized hardware is now dark silicon :)