As someone who has been on that side of things for 10 years, can we stop building really complicated stuff that requires this pile of shit and dedicated staff to run it.

The whole otel, logging and metrics thing is based around "record everything" and dig through the pile of shit looking for things that might be helpful. Fine until you have 50TB of logs/traces online which is our problem. Oh that and the 50 gig prometheus instance. 99.99999% of all information we record is entirely and utterly useless and costs us money

Make simpler things, log only exceptional things, use the rule of silence.

▲

otterley 21 minutes ago | parent | next [-]

I’m having trouble understanding your take here. Why is recording logs with embedded metrics at full resolution a problem, assuming you can afford to?

▲

bcvvvy 13 minutes ago | parent [-]

The latter is the problem. When your observability stack takes 20% of your cluster operation costs, 40% of your license costs and 25% of your salary costs, you’ve been licking toads.

Objectively we did the same job without it 15 years ago under the same traffic conditions.

▲

otterley 8 minutes ago | parent [-]

Those are completely made-up numbers. We don’t pay anywhere near that. About 10% of our cloud spend is on observability, and I targeted 15%.

That or you’re getting suckered.

▲

bcvvvy 7 minutes ago | parent [-]

Do you have a steaming pile of 1000 microservices?

	▲	otterley 7 minutes ago \| parent [-]
		Unfortunately yes, but we’re working to improve that :-)

▲

Fysi 15 minutes ago | parent | prev | next [-]

Resume driven development baby!

	▲	bcvvvy 12 minutes ago \| parent [-]
		Vendor PR driven development.

▲

BodyCulture 36 minutes ago | parent | prev [-]

How do you implement that? Would you like to explain, I am very interested!

How do you decide what is exceptional when you have only silence to compare to?

	▲	bcvvvy 12 minutes ago \| parent [-]
		When something shits the bed in your software, write a simple log out. Otherwise STFU.