Does edge inference really solve the latency issue for most use cases? How does cost compare at scale?

Depends on how much the latency matters to you and the customers. Most services realistically won't gain much at all. Even the latency of normal web requests is very rarely relevant. Only the business itself and answer that question though.