| ▲ | cortesoft a day ago | |||||||
In many cases, yet it is that simple. However, in other cases it can be more complicated. For example, if the requesters are other internal services, they might not be behind a single gateway (although you should then have other controls you can use). Other times, it is not easy to block the RIGHT requests, because it is hard to distinguish which backend services will end up being hit by a request (a single request can trigger multiple calls to multiple services). There is also the problem of catch up - if a system in a stable state is close to max capacity, it might never have the capacity to recover from a failure even if the entire system is restored. For example, imagine a log processing system that can just barely handle the normal peak load; if you have a failure where your producers locally store logs until they can send them upstream, you will have to clear that backlog, which requires additional capacity beyond a full steady state, which you might not have. | ||||||||
| ▲ | macintux a day ago | parent [-] | |||||||
And you never really get an opportunity to thoroughly test these failure & recovery modes until something catastrophic happens, at which point capturing detailed metrics and assessing the right sequence to perform the actions are both challenging at best. | ||||||||
| ||||||||