All of the significant server applications I have encountered in the industry have suffered from the same problem, which surprised their authors but seemed obvious to me: the application was spending the majority of its CPU time doing meta-work like entering and leaving epoll, stealing work from itself, etc. There are principles for writing Tokio servers and these are good points in the OP but I think they are little-known and too easy to violate.

I can't say I'm surprised when I see the 100+ function stack traces that Axum built on Tokio produces.

Before you say Axum is "holding it wrong" the project lives under the tokio-rs GitHub org.