Remix.run Logo
hurrrr 2 days ago

I've never run a bank either, but their current setup is very cost effective. A true multi cloud architecture would be "much" (a couple of 9s) more available, but I believe it would introduce other issues. The goal was to ensure the use of a subset of features for short periods of time, which is probably already above the market standard.

danpalmer 2 days ago | parent [-]

The cost effectiveness doesn't really add up for me either though (and again, I lack a lot of context). 1% of cost to run everything essential doesn't pass the sniff test, I assume it's much more during an actual failover. But I'm also not suggesting running any more hardware than the number of failure domains you want to be resilient to losing, and on cloud you could choose to run less and scale (assuming you aren't concerned about stockouts).

My guess is a little of both. I would bet that the 1% is the unused cost, but that for periods that they are running the stand-in system it's much closer to normal. But from what I've heard of their infrastructure I'd also bet that stand-in is significantly cheaper in part because it had the benefit of hindsight and could resolve issues in the first system.

None of this really changes my original feeling though. A cell based architecture spreads the risk, failover is essentially built-in without being a special mode you have to put the system into and then recover from again, on cloud you can scale those up and down with traffic and they also take peaks from each other, and so on. There's a reason why there are very few cloud outages that cross all regions for a provider, and if you're paranoid you can run multi-cloud (as they already do).

AlotOfReading 2 days ago | parent [-]

"Just do multi-cloud" is burying the lede a bit. Monzo is a bank, so it's not a given that they can easily partition all their operations into fully independent cells. I'm reasonably confident that teams of average competence could implement a second system for failover. I'm much less confident in teams of average competence doing complicated distributed systems work correctly, and doing it incorrectly is potentially existential risk for a bank.