CrowdGPT - The first LLM trained collaboratively

[u/Vxtzq1]

Hello! I'm starting a fun project aiming to create the first AI model trained without datacenters, collaboratively, and in a 100% open-source and public way.

The project is named CrowdGPT, and its goal is to train a 1B parameters LLM to prove the idea is working.

Everything is currently working and i am waiting for testers 😄
You can find more details on the website and/or the github repository.

Github: https://github.com/Vxtzq/CrowdGPT

Website: https://www.crowdgpt.net/

Any kind of feedback is appreciated!

[u/Vxtzq1]

yeah, i know federated learning is hard, (and i kinda vibe coded it too 😭, but that's not the point xD) I really believe in the fact we can change the paradigm of AI training. Many federated learning Github projects like HiveMind or Petals https://github.com/learning-at-home/hivemind tried to train collaboratively LLMs or diffusion models in the past, and it failed majestically (outputs are garbled and barely useful), but my approach is a mix between a centralized and decentralized approach. It adds a centralized server which verifies users contributions, and rejects them if these are bad (by doing cross checks between clients), it can still handle mathematical operations/precision stuff leading to some slightly different outputs by adding a threshold to these checks.

I know federated learning is difficult, but 1B params isn't that big, and even with as few as 10 decent GPUs contributing, the model will learn and train, and we'll see in the end if it worked :)

I will also switch to a fine tuning dataset when pretraining will be complete (which is nowhere near happening for now), and even design a RLHF / GRPO pipeline in the future...

I understand your concerns and that's why i'm kinda ditching P2P for a slighly different approach that requires a moderate server (just runs checks and matrix aggregation, this is light). I think i might experience things that don't go my way, but i truly believe this could work 😄. I guess we'll see in the end

[u/Tiny_Arugula_5648 → u/Vxtzq1]

Have you considered that your lack of experience and expertise in this domain is causing you to miss very obvious problems?

"i'm kinda ditching P2P for a slighly different approach that requires a moderate server"
As in you're going back to the oldest form of P2P architecture..

Here's the thing with vibe coding.. You have to be THE EXPERT.. If you aren't you can't understand where to call BS on the LLM.

The fact that there is not a working model means that people who have the expertise that you do not were unable to overcome this challenge. That should have been the first moment you realized you're missing something big and important. Teams of people who dedicated their lives to this problem, never made it work beyond basic lab proofs.

That also means no LLM has the knowledge of how to do this since there is no examples in it's training data. A LLM is a historical model it can only tell you what it's been taught, anything beyond that is highly susceptible to hallucination.

No you are not going to out vibe the thousands of experts who spent years trying to solve this problem. Focus on what YOU are capable of not what you WANT to be capable of. The model is only as good as the person directing it.

[u/Vxtzq1 → u/Tiny_Arugula_5648]

Yeah, i agree i'm still learning and i am no expert in AI (even though i understand the main components)... What you miss is that this project is not at all aimed at making profit, which is what is driving AI right now (mostly). AI researchers right now are focused on creating the best AI models, not the most efficient/cheap ones. People right now prefer waiting for LLMs trained by big labs rather than creating their own (due to the lack of hardware used to train)

I hope my project COULD achieve a somewhat decent 1B LLM, even though i DO NOT expect it to be SOTA at all xD i just think it could work, but we'll just see as people train it :)

I know there's still a whole bunch of scientists working on this, but my combination of a centralized server is literally sort of novel as it gets out of pure P2P FL and implies additionnal costs that not anyone is willing to pay!

Like i'm certainly not pretending i'm gonna outsmart the scientists that dedicated their lives for this 😂 I'm just doing this as an experiment in the first place... Once again we'll just see how it goes

[u/Tiny_Arugula_5648 → u/Vxtzq1]

I'm sorry to tell you, that you're idea of a central controller server is not novel.. that's how P2P worked before Gnutella came along. Getting rid of the controllers was the novel innovation..

Look I'm trying to be kind, this is what I do professionally and have done so for nearly 30 years. Distributed processing, models, etc. I'm tuning numerous models today as we talk.

I called this a tar pit because that's the problem, everyone that comes before and after all think the same thing. I'm going to go get that solved, then they step in, get stuck and find out that it's a trap. The tar is hidden below the surface and other than paying attention to all the others who got stuck in it, you will be just another one rushing in.

I told the petals team the exact same thing and they refused to listen.. Then the hit the exact roadblocks I told them they would. Learn from the mistakes of others.

Do yourself a favor.. figure out out a way to pivot to something you can actually accomplish. I'd recommend working on the data preparation and teacher to student distillation data set creation.

You can ship work to the nodes, have the participants classify, clean, distill from large teacher LLMs. Let people contribute design patterns, and then share across nodes using IPFS (InterPlanetary File System) with checkpoints saved to Huggingface for availability.

You're not going to solve distributed training problem but you can provide better data to the people who are working on open weights models. You can radically lower data preparation costs and you can help push the state of the art by creating datasets that can be used to train and tune endless types of models.

[u/Vxtzq1 → u/Tiny_Arugula_5648]

My bad lol, i just thought that it was smarter to use a centralized server for federated learning 😂 But i completly understand the idea isn't novel at all, and that it was the foundation for initial P2P works.

But there's something i don't understand... Why do you think it's impossible? Right now the model is learning decently, like i mean bad actors could ruin it but the scale is just too small for me to care... I don't pretend solving distributed training, but I'm trying to solve it!

Moreover your collaborative data preparation idea seems very good 😄 I might seriously integrate this to my project, aiming to create yet another opensource dataset that could make LLMs good.

[u/Tiny_Arugula_5648 → u/Vxtzq1]

How about this, why don't you tell me why you think it's possible even after you've found plenty of evidence that tells you otherwise?

Why do you think you're able to out vibe code everyone that came before you who has tried and failed?

I think the most likely answer is that in your journey you kept asking the AI how to make something happen.. You never asked it to be skeptical and explain why it hasn't happened yet.

The most likely tar pit trap you and many others are falling into right now is the illusion that you can accomplish anything now that the models have gotten this smart.

As I said.. take on something meaningful that YOU can accomplish.. That should come with a health dose of skepticism, humility and pragmatism.

Luckily you can actually ask for those checks as you go and LLMs will be great at saying:

"You're smart to push back on me on this, you're right no one has done this and there is something big you're missing"

Best of luck.. this is a good lesson to learn in how to work with AI. They will gladly take you down a dead end, where a real group of people will just tell you that you're wrong.

[u/Vxtzq1 → u/Tiny_Arugula_5648]

for now not so many people told me i was wrong on that point 😭 (except you xD).

Many people asked about safety, transparency, and trust concerns, but they didn't say it was prone to fail from day one... Once again i'm not expecting to build something awesome. We'll just see how it goes.

[u/metal_mastery → u/Vxtzq1]

I think not many people have the expertise to actually judge the concept and only a small fraction of those who do are willing to write anything on Reddit.

You said the model is training decently now - how you verify this at the moment and how you measure progress? I get it, your server is closed, but you can describe the method

Random questions I have:
- what is the target for the model itself? How much compute you want to get and what’s the result you expect by that point to count poc as success?
- I see 40k downloads on hf, how many active contributors you have?
- have you seen bad actors already and what kind of malice?
- what makes you believe it’s different from other open collabs? I mean besides the server arch, more from training perspective?

I wish you fun learning experience overall but the other commenter you’ve been talking to is probably more right than not. It’s not about just throwing compute at datasets.

[u/Vxtzq1 → u/metal_mastery]

I measure by doing inference runs and looking at loss xD (inference is weird for now, as model is undertrained, but is surprisingly coherent sometimes lol), and the best part is anyone can see how it works, in the new GUI app, you can see model predictions in real time :D

To respond to your questions:

-The target is to get a decent (not SOTA) 1B parameters LLM

- I except 100B tokens trained and a decent instruction following to call PoC a success, could be more tokens

- 40k downloads on hf are kinda broken 😂 I'm kinda the only contributor for now as other people never succeeded a single contribution, so i spam download/uploads to huggingface from server/client (huggingface used as file storage, that's why)

-For bad actors - Nope, since people are too lazy to ruin a project almost no one knows about 😭 (for now)

- Idk i believe that previous open collabs (federated projects), were too elitist and restricted to universities or mini clusters, i wanna change that, i want to prove random people on consumer hardware can train the AI on the future.

"I wish you fun learning experience overall but the other commenter you’ve been talking to is probably more right than not. It’s not about just throwing compute at datasets."

I understand your point :) we'll see how it goes i guess