I’ve managed a few 8 figure infrastructure bills. The exponential decrease in hardware costs combined with the exponential growth in software engineer salaries has meant that these bills become inconsequential in the long run. I was at one unicorn which had to spend 10% of Cost of Goods Sold on cloud. Today their biggest costs could run on a modest Postgres cluster in the cloud thanks to progressively better hardware.

100k/mo on cloud watch corresponds to a moderately large software business assuming basic best practices are followed. Optimization projects can often run into major cost overruns where the people time exceeds the discounted future free cash flow savings from the optimization.

That being said, a team of 5 on a small revenue/infra spend racking up 100k/mo is excessive. Pedantically, cloud watch/datadog are SaaS vendors - 100k/mo on Prometheus would correspond to a 20 node SSD cluster in the cloud which could easily handle several 10s of millions of metrics per second from 10s of thousands of metric producers. If you went to raw colocation facility costs - you’d have over a hundred dual Xeon machines with multi-TB direct attached SSD. Supporting hundreds of thousands of servers producing hundreds of millions of data points per second.

Human time is really the main trade-off.