My back-burner idea: the equivalents of Nostr relays, Mastodon instances, Discord servers, etc. needs to be self-hostable in client apps for decentralized social media to work.

Not only classical P2P software did exactly this, it worked/works beautifully. The only reason classical P2P software did not work and did not replace WWW as application was because of piracy prosecutions that makes users responsible for hosting data with unknown content that would be later revealed to users to have been pirated content that the user would be criminally responsible a la illegal substances through airport checkpoints.

That's bad - both the fact that users' passive actions are incriminating, and substances too. So the next-gen P2P would have to have plausible logging content filters such as AI-based CP and Hollywood detectors that would stop relaying of such content so that users can defend themselves in criminal courts. Or the systems could also be in-circle specific so nothing of value is lost if everyone in a terrorist group would be prosecuted together, though I imagine that could hamper network growth.

But fundamentally, I think the model has to be that all clients are also servers. That's the best way to decentralize a social media.

I think this is how iroh works. They have "relays" which are servers that help establish a connection between 2 clients if necessary.

https://www.iroh.computer/docs/concepts/relay

This is cool but P2P doesn't work. Iroh also relies on "relays" in a sense. Nostr makes that explicit and gives relays identities so they can freely enact policies instead of having to hack that in weird ways.