| ▲ | irthomasthomas 3 days ago |
| Yeah, I don't know exactly what at an AGI model will look like, but I think it would have more than 200k context window. |
|
| ▲ | doug_durham 3 days ago | parent | next [-] |
| Do you have a 200k context window? I don't. Most humans can only keep 6 or 7 things in short term memory. Beyond those 6 or 7 you are pulling data from your latent space, or replacing of the short term slots with new content. |
| |
| ▲ | sixQuarks 3 days ago | parent [-] | | But context windows for LLMs include all the “long term memory” things you’re excluding from humans | | |
| ▲ | kadushka 3 days ago | parent [-] | | Long term memory in an LLM is its weights. | | |
| ▲ | echoangle 3 days ago | parent [-] | | Not really, because humans can form long term memories from conversations, but LLM users aren’t finetuning models after every chat so the model remembers. | | |
| ▲ | esafak 3 days ago | parent | next [-] | | He's right, but most people don't have the resources, nor indeed the weights themselves, to keep training the models. But the weights are very much long term memory. | |
| ▲ | kadushka 3 days ago | parent | prev [-] | | users aren’t finetuning models after every chat Users can do that if they want, but it’s more effective and more efficient to do that after every billion chats, and I’m sure OpenAI does it. | | |
| ▲ | echoangle 3 days ago | parent [-] | | If you want the entire model to remember everything it talked about with every user, sure. But ideally, I would want the model to remember what I told it a few million tokens ago, but not what you told it (because to me, the model should look like my private copy that only talks to me). | | |
| ▲ | kadushka 2 days ago | parent [-] | | ideally, I would want the model to remember what I told it a few million tokens ago Yes, you can keep finetuning your model on every chat you have with it. You can definitely make it remember everything you have ever said. LLMs are excellent at remembering their training data. |
|
|
|
|
|
|
|
| ▲ | boznz 3 days ago | parent | prev | next [-] |
| I'm not quite AGI, but I work quite adequately with a much, much smaller memory. Maybe AGI just needs to know how to use other computers and work with storage a bit better. |
|
| ▲ | kurthr 3 days ago | parent | prev [-] |
| I'd think it would be able to at least suggest which model to use rather than just having 6 for you to choose from. |
