Remix.run Logo
sdcfgy 2 days ago

I suspect Monzo was bitten by complexity. The network should not be assumed to be reliable and if you make a thousand things (microservices) with network between them regardless of what you think may happen, it’s entirely impossible to test it with any level of certainty. And that’s worse if you have a complete failover scenario.

Add the instability of the average cloud control plane during an outage and things get rather nasty rather quickly in reality. We’ve been through that with AWS’ numerous outages.

My company is victim of this mindset. We sacrificed tested and assured reliability on physical hardware in multiple redundant data centres for cloud and microservices. Mostly driven by Monzo’s claims, semi religious thinking and me too.

Reliability is measurably down, cost is higher, latency is higher and we cannot fail over to another provider because someone went to an AWS conference and started using all the toys. Even if we did get rid of those dependencies, during an outage the kubernetes control plane is so tied into the vendor that we’d be fucked anyway. Last time the ingress controller couldn’t change ALBs for example.

Only winner here is Bezos and the decision makers CVs. The losers are the customers. We live in the stupidest of times.