> latent space representation of the conversation

and how would you load that back into the model? they are token-in, token-out, plus the KV-cache which is derived from token-in

They are not really token-in token-out per se, they are embedding-in embedding-out.

When operating on text, you embed each token into the LLMs embedding space. You go from a discrete token to a point in embedding space.

Likewise, when processing images, you have a image embedding model which produces a set of embedding vectors representing the contents of the image in the LLMs embedding (latent) space.

This same concept can be extended to compaction. Instead if limiting yourself to discrete tokens, you could generate a set of embedding vectors which represent the contents of the compacted conversation in latent space.

These have the possibility of containing a lot more semantic information per vector, which is why this can be appealing.

A big downside is decreased interpretability. AI safety people are generally fairly opposed to latent space reasoning for example, it can be harder to tell what the model is actually doing and if it is trying to deceive you.