I think this is cool, it's not too hard to get a 500m model doing some sort of interesting stuff as a home researcher but scaling it into the low billions to see if it remains competitive out of my own pocket is a non starter.

Exactly! Even if we could get to training 8b models that would be amazing