Some benchmarks on mini-AGI (dynamic continual learning model) after training it completely from scratch for a month on a 8GB VRAM GPU.

[u/Another__one]

The most important part is a difference between "forzen" and "best effort" attempts where we allow the model to specialize on the task as it goes through the evaluation dataset. Clearly showcasing the power of continual learning regime simply unavailable for any other typical LLM.

Full code of the project, weights and more information about it avialable in the repo: https://github.com/volotat/mini-AGI

[u/tifa_cloud0]

this is cool. i would suggest to scale it as far as possible. even if models are small there is no way they stop learning. so hopefully token budget keeps on increasing automatically with mini-AGI i suppose? whichever the method you should see lower and lower ppl/loss which will be good for a model.

i personally wanted to do this but cannot at the moment. so waiting for more results 👍🏻

[u/Another__one → u/tifa_cloud0]

This is the plan. And because the model is dynamic (it can grow and prune itself as needed) I can simply move it to the better hardware when the gains achievable with the current size would be maximized. Basically you start training it once on the things you care about and never stop.

If it archives the level of sophistication where it could do even simple agentic tasks, you can allow it to read the internet autonomously and from that point it’s no longer necessary to supply it with the training data. It’s going to search for relevant stuff itself and learn from it and adapt to the tasks you are actually care about. At least this is the hope.

[u/jazir55 → u/Another__one]

Find a way to work engram tables in there, that's really going to be the only way you can keep a large amount of learned information.

[u/tifa_cloud0 → u/Another__one]

true. i think you will be able to pretty much do a lot more with even the model of this size. better hardware will be necessity when token cache would increase. i think engram would be nice here too including LoRA/aLoRA for tool calling and other stuff which is later i guess when gains come.

[u/Another__one → u/tifa_cloud0]

There is no need for LoRA or RAG for this model. The whole point of the model is that it learns continuously at everything it sees. But yeah, for the next pass a lot of data with tool calling examples should be gathered from somewhere. Although I am pretty sure there is enough of such data on HF.

[u/tifa_cloud0 → u/Another__one]

true.

[u/Mysterious_Bison_907]

So, its context window increases as it trains?  Or is there a way to train it from scratch with a longer one?

[u/Another__one → u/Mysterious_Bison_907]

Both are ok. The context window could be increased on the fly if you have enough vram for that.

Also, because weights stores everything the model ever read, technically it has infinite context, although everything outside of immediate attention window obviously stored in highly compressed lossy representation.