>Overall, nesformer runs 150–290× slower than the real console.

I like these kinds of projects because it helps to show the massive compute requirement between LLMs and ordinary programs.

A while back I saw some people saying we would just have LMMs hallucinate an entire operating system every single frame. I guess technically it's now proven to be technically possible, but it would be thousands of times slower while requiring thousands of times more compute (a multi-million x difference from an ordinary OS!).

> have LMMs hallucinate an entire operating system (...) technically it's now proven to be technically possible

You're comparing apples to oranges there!

To hallucinate is arbitrary, you can use a 2B image param model, it will hallucinate an OS no problem* running real time!

What this post is about is completely different, which is actually emulating the complete system in an LLM - that is hundreds (or more) times slower.

*absulutely crappy quality tho, unusable OS in practice

This is a bit different though. It’s using transformer architecture to emulate almost at the circuit level. It’s probably even more inefficient than something that was trained to emulate a cpu, but with the advantage of actually being an exact emulation rather than learned approximation.

It’s a bit like when a human brain remembers facts, if you sat down and computed the hash values of the thing you were trying to remember, and traced out a DB lookup in your mind, it’d be a lot slower than just “oh yeah I remember that”.

> A while back I saw some people saying we would just have LMMs hallucinate an entire operating system every single frame. I guess technically it's now proven to be technically possible, but it would be thousands of times slower while requiring thousands of times more compute (a multi-million x difference from an ordinary OS!).

Gosh. That would be a very silly use of an LLM.

Bad use of any AI, even image/video models which are certainly less wrong than this. Stuff like DLSS show we're getting stuff that could generate a game's graphics frame-by-frame, but even though it's possible to have an entire game environment (not just graphics, everything) done by an AI, such things demo better than they play due to temporal coherency.

That said, having an AI make the code for each thing as you need it, might work somewhat OK eventually. Wouldn't trust it today (security, data integrity, and a tendency to cheat on tests), but eventually it could happen.

This is exactly like if someone implemented a CPU using nothing but CSS animations. Maybe a fun way to teach about turing completeness, but pretty meaningless otherwise.

Let's say you are developing a product, choose 1:

1) Have a single person vibe together a computationally inefficient solution in 1 month that runs transformers on some backend.

2) Hire a team to work for 2 years to design, perfect, optimize, plan, revise, and architect a solution.

The risk/reward for 1) is extremely better than 2). Given the turnover for entertainment in this day and age it doesn't make sense to do all this complex stuff.

That's the case right now--but what might a customized model tailored only for a narrow set of OS functions behave like? I suspect its achievable with a lot less compute.

Now is that a desirable product? Would it succeed on the merits in the marketplace? Maybe it would...

These systems in weights are proof of concepts. Surely there is room for optimization in there somewhere.

Isn't that the same with e.g. human brain? Computer program can compute billions of equations, whilst human will struggle with just one.

It would be a natural extension of the current work in software. Every layer of abstraction/web/scripting language adds seemingly 10x in both ram and processing time, undoing the advancement of 1000x faster processors and ram increases.

But at some point we’re hitting a wall here. If we want our software to undo 100,000x more processing powers, we need that llm software emulation layer.