> a lot of (software) engineering wisdom and best practices fails in the face of business requirements

They fail on reality. A lot of those "best" practices assume, that someone understands the problem and knows what needs to be built. But that's never true. Building software is always an evolutionary process, it needs to change until it's right.

Try to build an side project, that doesn't accept any external requirements, just your ideas. You will see that even your own ideas and requirements shift over time, a year (or two) later your original assumptions won't be correct anymore.

You can still design for evolution and follow best practices. That's actually IMO a hallmark of good software design.

The issue is when the evolution is random and rife with special cases and rules that cannot be generalized... the unknown unknowns of reality, as you say.

Then, you just gotta patch with if elses.