Already well documented that there is huge variance in inference serving efficiency; at every layer (model, software, hardware) you have opportunities for gains. Similar to how “the Chinese” haven’t undercut on cloud providers’ CPU offerings, there’s a very good argument that the hyperscalers will manage to keep their vertically integrated hardware/software edge here too.

I don’t really know where this commodification idea came from, maybe MBA types that don’t actually understand how cloud economics work, applying basic first principles analysis?

The operational engineering that goes into serving a frontier model is _highly_ nontrivial, it is not commodifiable. This isn’t a SPA application.

>The operational engineering that goes into serving a frontier model is _highly_ nontrivial, it is not commodifiable. This isn’t a SPA application.

I don't really see it frankly. Admittedly I never tried it (I only played a bit with local models) but conceptually how is it much harder than all the other cloud services that already exist? My educated guess is that building a competitor to Netflix, Youtube or Dropbox seem technically a lot harder to get running than offering inference on demand for instance.

It seems that the main barrier for competing in that market at the moment is getting your hands on the GPUs/RAM and a datacenter to put them in.

> Similar to how “the Chinese” haven’t undercut on cloud providers’ CPU offerings

No Western company will be allowed to choose a Chinese cloud provider by their legal department or by their stakeholders. Which leaves only the US cloud providers as viable sources (as us, Europeans, have been too busy building our revenues based on turism and other such non-tech thingies instead of building real stuff).