I think codebases and optimizations are a lot like this.

A lot of people seem to think the way to make things work better and faster is to add elaborate caching layers and layers and retries and GPUs and multi threading and...

I find the opposite tends to be true. Make things fast and reliable by doing as little as possible. If an API is flakey, make it not flakey, don't cache the result and add a retry loop.