While I think this is true, remember as we get more efficient we just decide to scale even bigger. So more GPUs, and more efficient.

I agree with the sibling comment, effiency is probably the more important component at this point. We are hitting not just a practical engineering roadblock for scaling with current technology, I think we have definitely hit a financial and logistical roadblock for up scaling with the number of GPUs (on an immediate basis)