Isn't this type of reasoning okay to an extent until you (or in more humbling cases, someone else) points out it's no longer working?

Sure. Occasionally there really is a problem in your signalling system. Even more rarely there's a fundamental design flaw.

If and when I see actually-doing-agile fail, I'll change my mind. But so far I've seen a direct correlation between the extent to which an organisation was actually-doing-agile and the effectiveness of that organisation, across a wide range of industries/environments/countries.