> With generated code you'll search for a long time

The observability people will claim that if the dynamic runtime behaviour of your system makes it hard to find the source of a behaviour, your system must be made more transparent and observable. They would also claim this was always the case -- we should never have relied on people's mental models being amazing because people move around.

(I don't know yet where I stand on this but I'm trying to learn more.)

If it was only "my" system without any integrations, I might agree.

But currently e.g. I am working on an MES/Scada layer that integrates data from a load of different machines in a factory. These machines are from China, Korea, Germany, Sweden ... Upwards there is an ERP integration (and some other systems).

Sometimes machines are updated and suddenly behave differently. Giving error messages in Chinese.

The ERP has the nasty behavior of returning error messages where it is not clear whether the actual processing actually happened or not. There are some heuristics on parsing the error messages, but these also change with new versions.

Sometimes one machine overloads cloud infrastructure and completely unrelated functionality fails.

Sometimes the on-premise network stops working for whatever reason and data is lost.

Sometimes operators do not understand a perfectly valid error message like: "The batch you loaded into input position XY has expired on XZ and cannot be used for production": "But we have been told to use it..."

So when you get called out at night, because the production line stopped and "MES is displaying an error message", it is mostly about finding out what integration failed and who else to wake up. Getting this right is very much appreciated by your colleagues.

And this is where you need a mental model of how things are connected, what error message happens because of what external causes etc.

Observability can only work perfectly for known problems. In a complex system for unexpected problem you can either provide too much data, so analyzing it and finding the relevant part becomes really hard, or too little data which makes finding the issue impossible.

There are so many companies claiming to provide the perfect observability solution and there are certainly solutions that help. But it is all very far from perfect.

Not relying on people is managers wet dream. And for a lot of people it might be true that they can be easily replaced. But for complex systems there are always some key people that you cannot replace without causing issues.

The observability people are correct. It’s not either-or though.