Sorry for the pretentious name, I know, I know.. It just contains all the pieces I would like to see a AGI model to have, and I can't stand the temptation. Before throwing rocks at me, please take a glance at the Readme, and I hope it will cover your mood a little bit.

So, first of all it does work and you can see the sample from the whole training run here: https://raw.githubusercontent.com/volotat/mini-AGI/refs/head...

Here is the scaling law graph I have so far, and it looks very promising: https://github.com/volotat/mini-AGI/blob/main/assets/scaling...

The model was built under my deep dissatisfaction so we cannot really train even moderately big models (1B+ scale) on the consumer's hardware. We can inference and fine-tune them for sure, but I would like to have full control over what the model sees over the training run, so it is fully aligned with my interests, not some corporations.

I was thinking about for some time and come up with two interesting ideas I thought worth pursuing: MoE with a lot of experts that gets added and pruned from the model while it trains, where only a small subset of of experts are actually in use at any particular moment + batch 1 training on the single continuous stream of data.

First allows us to be bounded only by the disk space in terms of number of parameters and load and unload experts only when they are needed. The second (if figured out and it turns out to be doable) allows us to get aways with small VRAM capacity because we do not need to store big randomized batches and their respective gradients.

I started brainstorming with Claude and after some time we found an approach that seems to be promising, and low and behold, a few weeks pass and you can see the results yourself.

Obviously, I did use AI in the process of making this project and I am pretty sure it would be completely impossible for me to do something like this without it, so I hope it is more than justified.

The model is still running over the first of 7.8B characters corpus I selected for training, so the weights are not out yet, and it's about a couple weeks of waiting until they are cooked at the current reading speed. And yeah, the model just read continuous interleaved passages from the dataset, each by 32K characters long each as a single stream. Just as you or I would do.

The set up seems to be really simple so you can git clone the project, run it and observe everything for yourself.

Thanks for your attention.

I have not looked carefully but it seems like this is over-promising on avoiding catastrophic forgetting.

The "trunk learning rate" is set at 0.1x the learning rate for the experts, so learning on different subjects disproportionately happens in the experts, and the trunk portion is comparatively more stable. But the population of experts can grow and shrink:

> The pool grows when it is short of capacity and shrinks when parts of it stop being asked for.

So:

- doesn't the trunk then _eventually_ still undergo catastrophic forgetting, it just may take much longer?

- and before that point, catastrophic forgetting happens in stepwise chunks whenever the expert pool shrinks?

This is an interesting approach. First of all, thanks for sharing your work. I've done. I want to say similar work in that I have trained continuous learning models and I have also offloaded parametric knowledge to hard drive people underestimate how difficult that is to do in a functional model. I look forward to digging in deeper.

This is pretty cool, thanks for sharing. Whe I read it first and saw "continual" I thought for a minute that it was implementing an idea I've been thinking about:

I want to have an agent that thinks continually/non-stop. Imagine a loop of "train of thought" that goes into the LLM and then out. Keep it going so that it "rumiates" thr way we do.

Then, add some sort of "messages" or IRQs when I want to communicate with it. To ask it things and whatnot. I think that sort of cycle in addition to this learning you are doing is what is missing for real AGI.

Nobody will throw rocks, I think most people are curious/suspicious about the big players and wants more hands-on since we suspect that this all will come down in cost soon enough.

If you actually scroll through the transcript he links to, you will see that something that looks like it could be training is happening, but no coherent responses are coming out at any point. At least not that I saw skimming through.

That might explain why there are no benchmarks of any kind.

Interesting. I wonder how much could be gained from using tokenization, which makes the model work at a semantic level rather than a syntactic level? I think it’s a force multiplier, but idk if it works here.

This is the first thing I see in my life that really looks like proto-AGI, it deserves its name.

I wish I could understand what a single graph in that nice graphic of graphs meant. No explanation for any vertical or horizontal axis. Looks pretty though.

Seems interesting, I've been messing with a lot of continuous learning approaches lately and it's cool to see something that's built from the ground up for avoiding catastrophic forgetting. Worth a clone for sure

Is this architecture actually able to generalize or is it mostly based on memorization? Have you tried some basic tasks that require generalization? e.g. number addition etc?

Getting conceptually closer to how the human brain works. Looking forward to more of this.

Seeing 'Mini-AGI' and '8GB VRAM' in the same sentence is a breath of fresh air. Maybe local AGI isn't so far-fetched.

What's the advantage of doing this, versus becoming good at context management and RAG? I always found trained knowledge unreliable, given that it is lossy by construction.

Have you tested what it remembers from early in the stream after a shift in the topics thrown at it?

Have you thought about making the whole thing "self-similar"? Every time I hear about MoE I think (and I know it's way easier thought than done) "why stay shallow"? I mean by that: would it be possible to extend/adapt the architecture so that an expert can be a previously trained Mini-AGI model? And recurse like this? Inuitively I would think some form of generalization could happen, as higher level experts (in the recurrence stack) would become sort of the "intuition" layer.

what character prediction rates are you getting on some unseen datasets?

The expert swapping architecture is very nice. Have you considered doing nested reinforcement learning where you use the nesting as a sort of low pass filter?

The concept is as follows: You train a critic to mimic the datastream and then you train against the critic instead of training against the data. The idea behind this is that the critic will memorize the training data so you do not need to store the full training data anymore. One of the biggest issues with current online stochastic gradient descent is that it is inherently a memory-less technique where the training data acts as the memory.

You can spin this further by going deeper with the nesting and then dropping the supervised critic. I forgot how to put it in words but the goal is that by having a model train against a critic of the critic, you can then drop the top level critic and instead use the mid level critic itself as your meta learning objective to train the actor against an unlabeled data stream.

Top level critic: learns to mimic the labeled training data via online SGD, then you add a simple hand written loss function to compare the predicted output with a given input. Basically you build a model specifically for distillation. Mid level critic: learns a reward function that mimics the top level critic directly but only gets to see the unlabeled training data and the result of the top level critic. Actor: The actor is exclusively trained against the mid level critic

Through this concept you end up with the existing training data stored as objective inside the mid level critic so you end up training not only against the latest data but also the already memorized data which should lower catastrophic forgetting. Of course at some point you might need to update the mid level critic again and to avoid that you might get away with just adding a very very wide Linear RNN / State Space Model / Mamba / Gated Delta Net as the middle critic (shower thought: use internal RNN states to represent LoRA vectors).

This is slop. 8M parameter dense model with context length 64 that you train on enwik9 in 2h will have 1.15 bpb. This model has 1.8 (bits per byte, lower is better).

THANK YOU SO MUCH. This is the missing piece.

throwing crumpled paper ball

Very interesting - I have a tangential question.

What motivated you decide to release this. OpenAI or Anthropic will just hoover it up, maybe scale it up and use it if they are interested.

You probably won't know if they do, and the chance they will give you something back is near zero. Why did you release rather than try to scale and build yourself?

(I've been working on some thing, not similar, but not dissimilar in goal - and I just can't get over the fact that tech will steal without giving back)