No, it's a great idea. Any systems engineer knows redundancy is good, even better if it's across providers. I see no issues with this from a system architecture point of view. As to how the implementation is done that's up to them to do right.

▲

danpalmer a day ago | parent | next [-]

Redundancy is great, absolutely, but there are multiple ways to do it and this is not necessarily the right one.

Which is better: 3 DCs for 2 DCs worth of traffic, running the same stack but isolated failure domains... or 4 DCs for 2 DCs worth of traffic, that work in completely different ways, where 2 are left cold until needed. I can see some advantages of the latter, but not a lot.

This also isn't about fault tolerance at the code level like in, say, avionics. In those scenarios having multiple distinct implementations makes sense, but that's very different, you run all paths at the same time and check they agree etc.

▲

hurrrr a day ago | parent | next [-]

I've never run a bank either, but their current setup is very cost effective. A true multi cloud architecture would be "much" (a couple of 9s) more available, but I believe it would introduce other issues. The goal was to ensure the use of a subset of features for short periods of time, which is probably already above the market standard.

▲

danpalmer a day ago | parent [-]

The cost effectiveness doesn't really add up for me either though (and again, I lack a lot of context). 1% of cost to run everything essential doesn't pass the sniff test, I assume it's much more during an actual failover. But I'm also not suggesting running any more hardware than the number of failure domains you want to be resilient to losing, and on cloud you could choose to run less and scale (assuming you aren't concerned about stockouts).

My guess is a little of both. I would bet that the 1% is the unused cost, but that for periods that they are running the stand-in system it's much closer to normal. But from what I've heard of their infrastructure I'd also bet that stand-in is significantly cheaper in part because it had the benefit of hindsight and could resolve issues in the first system.

None of this really changes my original feeling though. A cell based architecture spreads the risk, failover is essentially built-in without being a special mode you have to put the system into and then recover from again, on cloud you can scale those up and down with traffic and they also take peaks from each other, and so on. There's a reason why there are very few cloud outages that cross all regions for a provider, and if you're paranoid you can run multi-cloud (as they already do).

	▲	AlotOfReading 20 hours ago \| parent [-]
		"Just do multi-cloud" is burying the lede a bit. Monzo is a bank, so it's not a given that they can easily partition all their operations into fully independent cells. I'm reasonably confident that teams of average competence could implement a second system for failover. I'm much less confident in teams of average competence doing complicated distributed systems work correctly, and doing it incorrectly is potentially existential risk for a bank.

▲

ricardobayes a day ago | parent | prev [-]

In your Google scope this all works out however if you're a mature but not yet prevalent, neobank whose reputation rides almost solely on availability, I'd still say theirs is a great approach. Even in the unlikely event that AWS throws in the towel and has a hard failure, users can still enjoy a subset of functions without elevated latency. On top of it, it might well be that user data can never leave UK, so geographical redundancy might not even be a thing here.

And yes you are completely right a completely distinct implementation is probably a no-go, but I'd think you could deploy a similar codebase with feature flags, with some frontend differences. I'm speculating but pretty sure they didn't reimplement backend functionality for this.

▲

janpeuker a day ago | parent | prev [-]

Agree, I know nothing about their system but in terms of clear auditability and accountability of systems I agree it's great. Every subsystem failure or inconsistency can be clearly attributed to one or the other site/stack, and the stand-in could for example apply other rules (e.g. approve only smaller transaction amounts that are accepted by an incident-only set of risk rules).