> Is anyone able to explain why it's so much slower when solid state doesn't really care about the data location?

It does. Just differently.

E.g. a lot of SSDs nowadays cheap out and save money by using slower and poorer quality NAND + faster and high quality NAND cache. So random often misses the cache a lot more.