It was mainly the early wall street types that cashed in big. If it was used as suggested by satoshi, then you were using it as spending cash rather than an investment to sit on, in which case you shouldn't have made much money on it.

Don't forget about the pineapple fund : https://en.wikipedia.org/wiki/Pineapple_Fund

Also wall street never considered it seriously until a few years ago.