I could never get this about modern machine/deep learning or even the Transformers. Yes, it's not exactly rocket science, but when I see the data flow diagrams, it's not clear what is calculated in real time or multiple steps.

Is it really one big computation f(g(h(x)))?

Yes.

Each token prediction is one big function call. Then you just recursively generate more tokens until run out of context or the model predicts a next token indicating end of sequence. Technically the model outputs a matrix where the last row is a probability distribution, but I’m counting sampling from it as part of the chain. Hundreds of billions of dollars has gone into just making the function fatter and gradually changing pieces here and there.

Inference (the real time computation) and training (the computation you do when developing a model) are deeply tied by automatic differentiation.

In a way, you train by repeatedly inferring (forward), calculating a loss (how much your model sucks with its current predictions) and then improving the model by differentiating.

Take a look at Karpathy's micrograd repo to become more familiar with the process.

What's your distinction between real time vs multiple steps? All computation is done in steps.

Is it all one big computation? Its turtles all the way down.

It's all vibes.