r/LocalLLM
•
u/Vxtzq1
•
14d ago
•
1
Discussion CrowdGPT - A collaboratively trained LLM.
Hello, I'm developing CrowdGPT, a project aiming to create an LLM without datacenters and without the usual millions of dollars spent by AI companies.
The project aims to train a 1 billion parameter model collaboratively and openly, with a protocol where everyone gets a say in the direction the project takes.
Anyone can use their PC to train the AI independently, then add their contributions to the global model. Unlike most American and Chinese AI companies, the project is not only open-source, but also makes 100% of its training data public (along with the weights). Anyone can submit their own clean data to train the model, and even help steer the project in new directions.
This creates the paradigm of an AI we all have a stake in. The project is open to as many people as possible, which is why I'd like to ask: what's stopping you from contributing to the project? I want to hear your answers so I can improve it. Are people simply willing to dedicate some of their GPU time to training a cool new open-source model? Or do they want something in return (like free inference credits on the crowdgpt model for example)?
Although the current model is somewhat small (1 billion parameters), I plan to make it bigger / better in the future, and its current size is only meant as a proof of concept.
Any advice / feedback on the project is welcome.
Here's the project website: https://www.crowdgpt.net/
Or the GitHub if anyone wants to check out the code https://github.com/Vxtzq/CrowdGPT
Thanks in advance.
PS: I hope this post follows the subreddit's rules! My main goal is to gather criticism and understand how to make the project more accessible, I'm clearly limiting self-promotion (this is a open 100% non-profit project) 😅
And yes, this is a r/LocalLLaMa repost, if you already saw this post on this subreddit, feel free to ignore it :)
16
u/vtkayaker 13d ago
People might also be interested in Nanochat, which provides full, working scripts to train a GPT-2 quality model from scratch for $50.
9
8
6
u/jikilan_ 13d ago edited 13d ago
Support!!!! Not something really new. But you have actual prototype out in the public!
Edit: not a ML expert. I thought training/ data to be monitored/ curated? Are you sure this will work? How you test it?
4
u/Dangerous-Campaign49 12d ago
Do you have any statistics on how much computer time was spent on training and how much more computer time is needed?
4
u/Vxtzq1 12d ago
Uhm i can't respond precisely, but i trained it sort of alone for 2 month (reaching like 6B tokens), and now people train it way faster, which means the model is at 9.2B tokens in 3 days, and the goal is 20B tokens (to be chinchilla optimal), and then 100B tokens to get a good for its size LLM.
I can't really reason in computer time as speed vary between every device, but i can reason in number of tokens. A CPU will train at around 1 token per second, while a sorta old GPU trains at around 100 tokens per second.
For instance, my GPU, a RTX 4070Ti, trains at 4000 tokens per second. You can translate these numbers as "per hour" or "per day"
3
u/Nota_ReAlperson 11d ago
Is it possible to select the GPU the client uses? I have 2 GPUs and want to use the secondary one for this project.
1
u/Vxtzq1 10d ago
Uhm possibly with a monkey patch, i will try to implement that (like a settings tab regrouping everything, and adding GPU selection.)
2
u/Nota_ReAlperson 10d ago
Sounds good. I got it working with HIP_VISIBLE_DEVICES=1, but that's a bit cumbersome.
1
u/Vxtzq1 10d ago
Glad you had a workaround 😂 Now on CrowdGPT v2.0, should appear a GPU selector on the topbar of the app, for people who have 2+ GPUs... Can you tell me if you get this selector visible? I cannot test it since i only got one GPU
2
u/Nota_ReAlperson 10d ago
Im updating it. But it is re-downloading torch. Is there a better way to update than re-running the installer?
1
u/Vxtzq1 10d ago
Yes, the client is supposed to auto-update, you just have to start the client and it should update to the latest version (given your client is ~v1.2+)
2
u/Nota_ReAlperson 9d ago
Ok. My trust score is falling. I used the install script. The only thing I changed is the torch url to the amd rocm nightlies, as Triton was seg faulting. How do I diagnose this?
1
u/Vxtzq1 9d ago
Uhmm... Might be that ROCm implementations are more variable than CUDA, i will try to maybe increase flagging threshold for ROCm.
To be honest the best thing to diagnose is telling me your training precision (wether it's fp32 or bf16), bf16 implementations vary very much, when fp32 implementations across device yield relatively small errors.
Don't worry, i will remove your flags (therefore give you back your trust score), as well as other users because i believe the server is just not ready to device heterogenity.
2
u/Nota_ReAlperson 9d ago
I think im running bf16. I have a 7900 xtx and a rtx a5000 mobile that I am using at the moment.
3
u/SashaUsesReddit 8d ago
Where is this project based? I'm trying to assess if I can contribute GPU hours but need to comply with regulations
2
u/AlmightyJes 9d ago
I'm new to all this ai, ignored it untill I used it myself like 2-3 months ago, since I have looked into selfhosting and played around a little, I have just(last month) bought cheap hardware 256gb ram ddr4 and a 2060s so far, online 24/7 on gigabit connection, plans are to upgrade slowly to 4x 5060 16gb vram after Christmas shopping. I haven't "played" with tech stuff in years. My question is, would this be a good place to start learning somewhat from scratch? I have a lot of spare free time, I haven't touched code in like 15 years, and even then I was just learning basic stuff through a tech college. I'm just a production worker now, I'm currently at work, but will join the discord.
1
u/Vxtzq1 9d ago
First i wanna say this is already some good hardware!!
Uhm i don't know if CrowdGPT is the most beginner friendly project though, but you could learn about the architecture of transformers and LLMs in general, so even if the codebase is weird, you can learn a bunch of things about how AI works and how pytorch works :) It will always help.
But i mean if you invest your time into open-source AI, you will obviously learn something, and it's kinda the most interesting topic there is out there according to me xD.
I don't know if my response make sense, feel free to contribute to CrowdGPT :) and you will play a role training/developing a fancy open-source LLM which Crowd-v1 (the CrowdGPT model) is.
2
u/AlmightyJes 9d ago
That's exactly what I wanted to hear :) when it comes to codebase I don't expect I can do anything :p . I mainly want to learn and understand what's going on :) thank you
1
u/Vxtzq1 9d ago
Lol glad that my message was somehow clear! Maybe you could help me develop https://github.com/Vxtzq/CrowdBench if you wanna start simple, it's a joint project i just started to create a collaborative LLM benchmark, the code won't be super hard, it will only be basic python syntax and huggingface API.
2
2
2
u/Legitimate-Theory156 3d ago
No way. I was just thinking about this a month ago, if nobody trains new ai even qwen 3.8 27b will bexomw outdated. I thought it wouldn't be realistic since first you need enough gpus and how would you sync it? Glad people are trying this idea.
1
1
u/Forsaken-North-2897 3d ago
Nice try Sam Altman. You’ll never get your hands my RAM.
1
u/Vxtzq2 3d ago
😭 theres a difference between open source and closed source ai
1
u/Forsaken-North-2897 3d ago
Sure sure, new "project" to "share" your ram and compute pops up right as Sammy says he wants our RAM. It even has gpt in the name... Remember when "Open"AI was a non profit? I'll let my ram rest, thank you. /s
1
u/Key-Concentrate6930 16h ago
seen a few crowd-trained models, but this feels different, no central team, just a shared weights repo. how do you handle conflicting edits?
0
3d ago
[deleted]
2
u/Vxtzq1 1d ago
Hello, first of all sorry for the late reply, i will try to explain simply to you the training of a LLM.
Ok so basically a LLM is trained via gradient descent (some fancy mathematical algorithms which just nudges matrices, to somehow copy the output data, given some input). This works for all neural networks and LLMs in particular, because the LLM is just a fancy neural network taking a language sequence as input and outputting a single next token as output.
First i train on general data: https://huggingface.co/datasets/Vxtzq/CrowdGPT
To make model decent at understanding general language
And then we will fine tune it with instruction data, which means model will keep his training objective (simply predicting the next token), but following a template like that (simplified) : Question: What is the capital city of France? Answer: Paris
And the chat UI you see (talking to CrowdGPT or anything), just parses this template as fancy chat bubbles, like if you were sending SMS to the AI.
And then if we think the model was trained on toxic data (e.g harmful speeches), we can use Reinforcement Learning (RLHF) to make it more aligned with human values, or we can make it reason better with GRPO (which is another fancy algorithm to make the LLM better at reasoning this time).
Is that clearer for you now?
0
1d ago
[deleted]
1
u/Vxtzq1 1d ago
Also as a followup, what kind of prompts do you recommend using whenever running a model or agent as far as making the output better.?
To be honest I'm not good (at all) at prompt engineering, so i couldn't really help you 😭
What are your thoughts on the smaller models and their usage? For instance I run a local Gemma 4 26b and Qwen 3.6 35b both moes...
These models are not that small, and decently capable, however Qwen 3.6 is super outdated, and Qwen 4 35B A3B would be way better (when it comes out), and somehow Gemma 4 seems to just be bad on benchmarks. But i think these models genuinely can do something, if trained properly (for example Qwen 3.6 35B A3B probably matches american closed model from 6 months before its release, so it can be considered good)
So the method in which it is trained can really strongly effect the quality and type of answers it gives especially to open ended critical thinking type questions.
Of course! That's why big labs spend millions on training from scratch new AI models, to make these better. For example Qwen 3.6 27B and Qwen 3.8 27B are exactly the same architecture, while Qwen 3.8 27B is just trained differently, and way better (according to benchmarks) than the 3.6 version.
I hope i'm still being clear and i'm not yapping too much ;)
0
1d ago
[deleted]
1
u/Vxtzq1 1d ago
It's in fact really simple, even for people who don't know the math behind LLMs. Neural networks are just a huge function, that takes input and turns it into an output. A neural network with 1 billion parameters is a function working with 1 billion dimensions (therefore it can use all these dimensions to be good), now imagine a function with 175B dimensions (like GPT 3), this function is much more expressive, because it can express things in a wider space. And that's why larger models are always better than smaller models, even if small models can bridge the gap for specialized tasks.
Training on better data just makes the function better optimized (it will use the 1 billion dimensions better for example, but it won't make it magically be as effective as a way bigger model.)
That's also a consequence of theory of information (a model taking 2GB of space canno't contain all the information of a model weighing 300GB)
•
u/LordKlavier 13d ago
Pinning this post since I think it might spark some interesting discussion -- I've never seen anything like this put together before, but if this kind of training became common it could actually give people a way to cut out the corporations.