| ▲ | You could have invented transformers from memory(alexwang.ai) | |||||||
| 41 points by speckx 3 hours ago | 2 comments | ||||||||
| ▲ | craftkiller an hour ago | parent [-] | |||||||
This article does two things which make it unhelpful for me: 1. It uses math notation. This is a software development site, wouldn't this be much more clear to this audience written as psuedo code instead of a complex series of greek and latin letters? 2. It doesn't step through a concrete example. What is hollow R raised to the (M * d_v) ? Is it a number, a set, a list, a matrix? To use an example I understand: RAG used to seem complicated to me when only math people were talking about it in math words. For example, let's take a line from wikipedia on embeddings: > the representation is a real-valued vector that encodes the meaning of the word in such a way that the words that are closer in the vector space are expected to be similar in meaning What made it clear was learning that generating an embedding is just a function(string) -> list[float], and RAG is just taking a list[float] for your prompt and finding other list[float] that are close to it in your dataset. So what would make this article more clear, is step through a simplified example of putting a value into a memory store and querying that memory store, showing the actual data structures. | ||||||||
| ||||||||