But GPUs are a depreciating asset - if there's a bubble burst and your 5 million GPUs are idle for the next few years before demand picks up again, they'll be pretty outdated and of limited use.

Infrastructure tends to have much longer lifetimes. A lot of the telco infrastructure "overbuilt" during that boom is still used today - you can always blow new fibre, replace endpoints and all that without digging everything up again, which was the largest cost in the first place. Sure, in the above example you'll still the datacentre itself (and things like electricity connections and cooling) that can be reused, but that's a relatively small fraction of the total cost comparitively.