Every 6-12 months or so we get an increase in one or more of things like: compute power, compute efficiency, GPU power, GPU efficiency, network bandwidth increase, memory speed increase, component density increase in the same form factor, etc.

For awhile it was every 2-3 years you'd start a hardware refresh. As companies moved into more and more training, this timeframe started to shrink. It went from 36 months to 24 months. From 24 months to around 16-18 months. Last I checked last year, it was at 12 months. I think things may have slowed because of component availability, but otherwise whole data centers would be 6-12 months into full operations before they would start a refresh cycle.

Not to mention the massive increase in power density demand and cooling demand per rack that entails.

So no, "AI costs" have not gone down, in fact they are more expensive on training AND inference than ever.

This is why many are concerned about the heroin drip of api costs into orgs. For the companies that are public, look into their financials. It's gonna hit companies and high volume users like a ton of bricks.