What you're missing is that for these massive systems there's never enough capacity. You can go look at datacenter buildouts YOY if you'd like. Any and all compute power that can be used is being used.

For individual services what that means is that for something like Google Search there will be dozens of projects in the hopper that aren't being worked on because there's just not enough hardware to supply the feature (for example something may have been tested already at small scale and found to be good SEO ranking wise but compute expensive). So a team that is able to save 1% CPU can directly repurpose that saved capacity and fund another project. There's whole systems in place for formally claiming CPU savings and clawing back those savings to fund new efforts.