It literally is token prediction with vector search.

Yes there’s an app layer in the chat product for convenience and parsing but the model is exposing methods like ‘complete’ that predict out word sequences.

When model temperature settings are not added in, you get the same exact response every time, just like ngram.

What you’re seeing with Gemini and ChatGPT is context caching to prevent getting sidetracked and response boilerplates with multi-modality so they can call out to image generation, a code completion service, etc. to assemble a complete response.

That’s why the mainstream providers seem so much better (or at least consistent in replies) - each prompt is actually requesting multiple models and doing a lot of application level work to assemble the response you see. That also why they take sooo long to respond.

The language model itself is quite literally a text completion machine, with settings.

The language model itself is a static data structure. On top of that is an inference engine. Currently all inference engines are autoregressive. Meaning they use the current output to produce more output. This isn't necessarily the only way to use a language model. Yann Lecun's JEPA is probably the most publicized project to work a different way.