A continuing, probably unending, opportunity/tragedy is the under-appreciation of representation learning / embeddings.

The magic of many current valuable models is simply that they can combine abstract "concepts" like "ruler" + "male" and get "king."

This is perhaps the easiest way to understand the lossy text compression that constitutes many LLMs. They're operating in the embedding space, so abstract concepts can be manipulated between input and output. It's like compiling C using something like LLVM: there's an intermediate representation. (obviously not exactly because generally compiler output is deterministic).

This is also present in image models: "edge" + "four corners" is square, etc.