How usable is this in practice for the average non AI organization? Are you locked into a niche ecosystem that limits the options of what models you can serve?

Yes, but in principle it isn't that different from running on Trainium or Inferentia (it's a matter of degree), and plenty of non-AI organizations adopted Trainium/Inferentia.