> A while back I saw some people saying we would just have LMMs hallucinate an entire operating system every single frame. I guess technically it's now proven to be technically possible, but it would be thousands of times slower while requiring thousands of times more compute (a multi-million x difference from an ordinary OS!).

Gosh. That would be a very silly use of an LLM.

Bad use of any AI, even image/video models which are certainly less wrong than this. Stuff like DLSS show we're getting stuff that could generate a game's graphics frame-by-frame, but even though it's possible to have an entire game environment (not just graphics, everything) done by an AI, such things demo better than they play due to temporal coherency.

That said, having an AI make the code for each thing as you need it, might work somewhat OK eventually. Wouldn't trust it today (security, data integrity, and a tendency to cheat on tests), but eventually it could happen.

VibeOS? It's been done! It's hilarious: https://youtu.be/7NfyZhV1dKM?is=Y8LwTbY_stLPilqR

> Stuff like DLSS show we're getting stuff that could generate a game's graphics frame-by-frame

A key point of DLSS (5) is that it is significantly guided by ground truth rasterization or raytracing info. It's not intended to replace graphics pipelines, it augments them to produce more detailed surfaces and interpolate frames more efficiently than calculating all of it traditionally is. But you have to calculate some of it to get these results, it can't be a purely AI solution without looking incoherent.