You do 10 things at a small conference, everyone says "we need more of X{0}..X{9}", you have more things next year, more people, everyone wants more of whatever, more people, more problems with more people (security, cost, sponsors,..), more attention of mainstream media, more people next year, more push for politics, more people, more issues with more people, etc., and in the end, you get a boring business conference like many others.

I'm pretty sure that each of the niches could make their own conference now, at some small venue where a 100, 200, 500 people would come... SNES hacking and development? Sure, a small, really nice conference... but then someone would want NES too, and N64, and sega, and PS1, and corporate sponsors, and you end up with E3 instead of 50 retro developers and 150 curious people doing interesting stuff.