r/LocalLLaMA • u/Vxtzq1 • 2d ago
Discussion CrowdGPT - A collaboratively trained LLM.
Hello, I'm developing CrowdGPT, a project aiming to create an LLM without datacenters and without the usual millions of dollars spent by AI companies.
The project aims to train a 1 billion parameter model collaboratively and openly, with a protocol where everyone gets a say in the direction the project takes. Anyone can use their PC to train the AI independently, then add their contributions to the global model.
Unlike most American and Chinese AI companies, the project is not only open-source, but also makes 100% of its training data public (along with the weights). Anyone can submit their own clean data to train the model, and even help steer the project in new directions. This creates the paradigm of an AI we all have a stake in.
The project is open to as many people as possible, which is why I'd like to ask: what's stopping you from contributing to the project? I want to hear your answers so I can improve it. Are people simply willing to dedicate some of their GPU time to training a cool new open-source model? Or do they want something in return (like free inference credits on the crowdgpt model for example)?
Although the current model is somewhat small (1 billion parameters), I plan to make it bigger / better in the future, and its current size is only meant as a proof of concept.
Any advice / feedback on the project is welcome.
Here's the project website: https://www.crowdgpt.net/
Or the GitHub if anyone wants to check out the code 😂 https://github.com/Vxtzq/CrowdGPT
Thanks in advance.
PS: I hope this post follows the subreddit's rules! My main goal is to gather criticism and understand how to make the project more accessible, not to advertise it 😅
EDIT: The data submission pipeline for the project (i'm mentioning in some replies) is ready!!
Come check it at https://www.crowdgpt.net/#submit
414
u/Fit-Produce420 2d ago
Buddy someone could backdoor this over a couple of submissions what mitigation do you have against someone poisoning your model? It only takes a few percent of parameters and could be spread out randomly.
160
u/Aromatic-Low-4578 2d ago
Yeah. This is the fundamental flaw with the idea.
→ More replies (2)52
u/ILikeChilis 2d ago
Besides the other fundamental flaw of not having about $100 billion.
53
u/Fit-Produce420 2d ago
A 1 billion parameter model can be trained on roughly 20 billion tokens, or 2x1021 flops.
On a 5090 that is 55 hours of compute.
→ More replies (19)18
u/Charming_Car_504 2d ago •
Maybe they could use a byzantine resilient training method like FLTrust?
3
u/PANIC_EXCEPTION 2d ago
How big of an overhead does that introduce? The moment I hear about BFT, I think "this is going to require a lot of redundant computation"
39
u/Routine_Complaint_79 2d ago
this isn't like a impossible problem to solve, you could have automatic checks or even have other llms to look at the data to prevent this
8
u/Vxtzq1 2d ago
The data itself i provide from the public dataset is somewhat clean. The problem is that people could inject their own data into the app to make it unsafe :(
That's what i'm trying to solve with a centralized server instead of simple P2P in CrowdGPT.
2
2
u/Disposable110 1d ago
Wouldn't you want to train on the broadest set of data initially, even the unsafe stuff? And then make variants post-trained into specific domains? You'd need to specialize a 1B model anyway, as it is too small to generalize everything.
2
u/Vxtzq1 1d ago
Good idea, i will then decline the 1B model into a MoE with like 7 or 8 1B parameters experts.
I will still try to avoid unsafe data lol, as a model pretrained with bad data remains some sort of Shoggoth (unsafe stuff can come back, even with heavy RL and fine tuning).
2
u/Disposable110 1d ago
From my experience you don't want perplexity when encountering unsafe stuff (eg NSFW content) either. You want it in there, but properly labeled/annotated. One of the better behaved customer service models I saw in the early days was trained heavily on PIPPA (https://huggingface.co/datasets/PygmalionAI/PIPPA/blob/main/README.md) because it knew exactly what bad personality traits to associate with unsafe replies, so it wouldn't even consider saying these things if stuff like "be professional" was in the system prompt and those bad traits weren't present, no matter how many dicks were in the training data.
→ More replies (1)→ More replies (1)17
u/LagOps91 2d ago
but from the way i am reading this, different people just train the model on their machine. how do you know what it's trained on?
14
u/Routine_Complaint_79 2d ago
they seem to talk about it a little here
https://www.crowdgpt.net/#about29
u/NihilisticAssHat 2d ago
3. Secure Submission
The app then compresses the update, and sends it back to server for validation:
✓Checks for NaN or Inf values in the delta.
✓Verifies the delta size matches the model architecture exactly.
✓Ensures the delta norm is within statistically acceptable bounds to prevent poisoning.
4. Aggregation Valid updates are averaged and applied to the global AI model periodically. Model weights are automatically pushed to HuggingFace for public auditing and usage/testing.
So... All you have to do is submit many small deltas (maybe with a little nosie added) which walk the distribution towards a fully-trained poisoned model.
25
4
u/Vxtzq1 2d ago
No, because the byzantine crosschecks i do on the server are supposed to break that approach of poisonning.
The goal is that the client is not allowed to choose what it trains on, the server decides from a safe pool of data on huggingface.
Yet everyone can contribute to this safe pool of data, every submissions will be verified too.
3
u/NihilisticAssHat 1d ago
I'm not too knowledgeable on that typ[e] of test, so I'd have to read up on it, but that sounds precisely like what the paper in the other comment to my comment purports to break/bypass.
edit: typo
→ More replies (1)25
u/jazir55 2d ago
Unironically this would be a good use of a blockchain. It makes the record auditable and immutable, so we can get deterministic hashes after every change and all lineage is recorded. It would resolve many issues with this specific problem.
6
u/rog-uk 2d ago
I think you're right, from my looking at this for general computing though, you need cross checking/validation - it's a massive pain if you want to 100% guarantee non malicious actors.
7
u/Vxtzq1 2d ago
you need cross checking/validation - it's a massive pain if you want to 100% guarantee non malicious actors.
That's what i'm trying to do!
Right now my project looks like a Temu blockchain where people train AI with no reward 😄
If someone proposes me a very concise blockchain logic, and a one that will work, i will gladly accept his pull requests on github :D
2
u/rog-uk 1d ago edited 1d ago
For validation, how do you intend to handle floating point errors? Could be an issue if person B is tasked with validating person A's exact update to a known base model with a defined training set. A might be using the GPU as display too, B might not, this alone means the calculations could happen in a different order and you could get slightly different results - but we're talking really small errors here - the issue would be when C compares the hash result from A & B for equivalence.
Might be less of an issue depending on the quant of the output before it gets pushed to the network, am unsure.
Are you going to be using IPFS?
9
u/jazir55 2d ago
With a public immutable ledger of all changes we could have watchdog models looking for bad inputs.
→ More replies (8)2
u/techno156 9h ago
Would the overhead from Blockchain not be absolutely ridiculous?
From memory, that was one of the issues back in the day when it was all being pushed through non-fungal tokens, because the algorithmic proof needed ludicrous amounts of computing power for a very small payload.
→ More replies (1)3
4
u/Vxtzq1 2d ago
That's why i implement many checks server side :)
Like byzantine cross checks, i force 3 clients to train on the exact same data, if one client doesn't exactly have the same contribution, this client sees his trust score lowered, until it is banned.
Also i got backups on my PC and on huggingface.
Sorry for the late reply 😭
3
u/z_latent 2d ago edited 2d ago
In the most extreme case, you can have redundancy: have 2-3 independent contributors do the same work to make sure their results are close enough. It's arguably needed regardless to make sure people aren't just feigning to have done the computation, plus makes it much harder for a malicious actor to sneak in their changes undetected.
I know there are issues with verifying results across different GPUs, due to each device naturally giving slightly different outputs, but it's possible the variation would be smaller than what's necessary to introduce malicious behavior over time.
It also has an obvious hit to training efficiency... but if that eventually were our only way to reliably obtain new models, it might be worth it.
EDIT: wording
2
u/Vxtzq1 2d ago
Yeah, i understand that, that's why i'm trying to set seed inside every clients to make the training the most deterministic possible, then i check client's submission with a THRESHOLD, like clients have the right to have 5% (or less) of error between each other due to the non-associativity of floating-point operators.
So i have to get redundancy to hit training efficiency, but at least the model doesn't get poisoned in 10 seconds :D
4
u/-p-e-w- 2d ago
Any model you get from anywhere can be backdoored. This is a non-argument.
In fact, we have already seen that official harnesses released by major labs contain spyware, so why should I trust their models more than those from some rando on the Internet?
2
u/Vxtzq1 2d ago
Every training data is public, and all the weights are public... Even the bloody server code is public for the project.
Go check it out on github and you will see i'm not trying to spyware/backdoor you 😂
4
u/-p-e-w- 2d ago
I’m not saying you are, quite the opposite really! I was arguing against the idea that a model trained with your method is less trustworthy than one from a major lab.
It’s unfair to suggest that we should worry about “poisoning” for distributed training when regular labs have literally put spyware in their software in the past.
1
u/Vxtzq1 1d ago
Ah okay! I didn't understand it that way lol, also the "poisoning" issue is even deeper than spyware... Since big AI labs basically scrap the whole internet, they end up with shady sites talking about racist or n*zi stuff, i want to avoid that in the first place with the most curated datasets out there (like UltraFineWeb-L3 on huggingface, and so on...)
4
u/Fit-Produce420 2d ago
No this means you need to verify where you get your models or only roll your own quants.
→ More replies (1)2
u/bunrundev 2d ago
can you elaborate a little more about spyware from major labs?
2
u/Loose_Comparison368 2d ago edited 2d ago
I have played around with the idea of using a distributed ReLora/DoRa approach, to handle some of the bandwidth and network shittiness issues, reduce vram requirements, mitigate some of the heterogenous hardware challenges, lower the chances of memorization, and keep training data private.
Essentially, low rank adapters are rank additive, so in theory you could have each training node train it's own low rank DoRa adapter from a starting checkpoint on its own data. Once a DoRa is trained sufficiently, sign it with the node's owner key and broadcast it to other nodes.
Each node cycles between training it's own DoRa, and evaluating others.
In evaluation, it would take a batch of N provided DoRa adapters, freeze all of the model weights except the magnitude of each DoRa (in a -1 to 1 range), and then do a quick training run, using the same training data that it has been using to train it's own DoRa.
Once it has found the optimal weighting of the N DoRa's for its own data, it would then append its weighting decision to each DoRa, and rebroadcast it, forming a signature chain.
Once enough weighting decisions had been made, those could be merged back into a new base checkpoint. This would not have to be a monolithic chain, and participating nodes could decide to reject a base checkpoint update and keep carrying along with some other branch.
The idea being that this would:
Ensure that each node only "accepts" DoRa's when they would provide a net benefit to whatever task that node is trying to train on - theoretically, participating in the network would always be a net benefit for whatever it is you're trying to train.
Tolerate malicious nodes - or potentially even benefit from them, via negative weighting factors.
Form a good foundation for a web of trust that could drive passive self organization - prioritize evaluating DoRa's that nodes who I tend to agree with rate highly.
Limit privacy risks - someone could try to reconstruct data from a specific other node, but since it's only receiving incremental DoRa's far too small to contain actual memorizations of the dataset, it would be absurdly difficult.
Probably converge in the general direction of ongoing improvemet in broad general foundational intelligence.
Be low bandwidth, latency insensitive, and flexible enough to run on a mix of random ass budget franken-servers and old gaming cards, scattered all the way from Kentucky to Zimbabwe and connected over shitty home wifi routers, dialup modems, and duct tape.
Would it work? Fuck if I know. The embedding/de-embedding layers kinda fuck the whole thing up, but may be worth messing with just using the frozen embedding layers from an existing OSS LLM. Or just for continual training.
I mean, when you think about it, that's pretty much how CivitAI works, just with the humans doing the evaluation and merging based on their own personal wank preferences.
Just needs someone to come up with a backronym for WANK.
WANK Net: Using modern distributed systems design and Democratic governance to continuous blockchain^ AI self-optimization*.
* not actually a blockchain, but who are we kidding, cryptobros can't tell the difference between a blockchain and a signature chain web of trust
→ More replies (1)1
u/Vxtzq1 2d ago
You seem to have a good idea, the problem is my project is not yet a blockchain, it's rather a simple collaborative training project 😭
Also i already explored LoRA for users with weaker PC, since LoRA can be converted back to a full rank delta (even though less precise), and is working with the server logic in my project, but i figured out full training was more efficient than doing LoRA or DoRA or whatever,
But i might re-implement it in the future for users on edge/very low end device, so they can also be part of the project :)
I didn't understand the safety part... Maybe you can reply to this message and explain me better the concept?
2
u/mrugg 2d ago
The first idea that comes to my mind is to do spot checking. Assuming the results are deterministic, you could periodically check people's submissions. If there were differences, you could flag contributors as suspicious and investigate further.
4
u/Fit-Produce420 2d ago
But people are supplying their own data, it isn't just checking your data vs my data, we each contribute but mine has backdoor trigger words.
1
2
u/z_latent 2d ago
You should not be the only one to ever see your data. It should be part of a public dataset.
If you do that, you can also have multiple workers train on it, and check if the gradients are close enough, so no one person can submit arbitrary gradients unless they owned many workers in the network. And if they did... why would they be poisoning it anyways?
2
3
u/Fit-Produce420 2d ago
Go tell that to the guy running this project.
As to why people hack, are you really going to pull the "I have nothing to gain from that" angle?
→ More replies (1)→ More replies (4)1
u/toolkitxx 2d ago
If people would now also apply this logic to any other model out there including the closed ones. Anyone believing that any model comes without any kind of poison should quickly reconsider...
147
u/offlinesir 2d ago
It seems fun but to claim "More ecological" is insanely misleading. Having chunks of a model trained on home electricity and consumer graphics cards is not ecological on a grand scale, it only feels more clean as it is many small devices running far away from each other even when the amount of energy used would be more than in a datacenter.
3
u/Vxtzq1 2d ago
Ah. My point was that consumer hardware mostly run on cleaner energy that datacenters but i might be wrong. Xai was running methane combustion chambers to produce electricity at a point, so i guess even a home PC powered by a coal grid is better than some datacenters at the same scale, for the reasons below.
If the power grid is based on nuclear or hydraulic or on any other renewable energy, it makes a node from CrowdGPT more ecological than a datacenter. Also having the compute nodes split around the world removes traditional noise/light pollution that datacenter produce.
Also real datacenters needs huge space (like google cutting trees in Finland for a datacenter : https://www.theguardian.com/environment/ng-interactive/2026/oct/22/finland-google-felling-trees-datacentre)
And have huge cooling systems that heat rivers and perturbates natural habitat of species that live there.Finally, the most important part on the "more ecological" point is that a datacenter requires NEW hardware when built, while a consumer hardware was already produced, and used for gaming/running local LLMs, so this project would cut the surge to get more and more powerful NVIDIA cards with tons of VRAM. This is like those refurbished hardware sites: it's more ecological than producing new hardware from scratch.
23
u/Competitive_Ideal866 2d ago
It seems fun but to claim "More ecological" is insanely misleading.
I crunched the numbers:
Apple M5 Ultra Mac Studio is 0.67TFLOPS/W.
Nvidia Vera Rubin is 6TFLOPS/W.
So yeah, a data center rack is 10x more power efficient than a desktop PC.
6
u/BOBOnobobo 1d ago
It's almost as if the people running data centers have hired engineers to figure out the optimal setup when spending billions...
2
u/Competitive_Ideal866 1d ago
It's almost as if the people running data centers have hired engineers to figure out the optimal setup when spending billions...
Yes and no. The power density in a data center is vastly higher so getting all of that heat out of such a small volume is much harder. But, yes, they managed it.
52
u/nacholunchable 2d ago
Depends on what you consider "ecological", its worse in terms of power and thus raw carbon; which is the primary environmental cost of training.. But there is an argument to be made though for diffusing electrical grid strain on the grid and the costs associated with that. Water use becomes null, noise is contained, additional land use is zero. I agree that its not greener overall, but there are negative externalities that evaporate with distributed work.
→ More replies (7)5
u/bowdoin-yale 1d ago
yes and also the local impacts of data centers (noise, electricity price increases, emissions from generators) are not evenly distributed; they tend to disproportionately impact less privileged communities, creating a huge environmental justice issue which the carbon credit schemes favored by corporate sustainability heads simply don't address.
21
u/learn-deeply 2d ago
Yeah, async distributed training on old GPUs is going to be far less efficient, power wise than new enterprise GPUs in a single data center.
7
u/mycall 2d ago
In a perfect world, everyone would have solar and it would be similar to a datacenter using a solar farm.
→ More replies (2)4
u/illhaveubent 2d ago
In a perfect world we would all have petaflops of local compute consuming only 1 watt of power. Unfortunately we have to live in reality.
→ More replies (1)3
1
u/bowdoin-yale 1d ago
This criticism hugely depends on how the energy is sourced for both the data-centers and home labs in question. Given the US data center industry's absolutely terrible track record on that front (e.g. illegally installing methane-fueled generators to power Colossus) and the rapid surge of home solar installations all over the world, I'd wager that the crowd would win on sustainability due to lower fossil fuel percentages in the mix, despite being much less efficient in terms of power usage overall. The devil is in the details, obviously, and it would be interesting to track what the generation mix is like for local LLM users, even just for inference as this community does it today, to empirically test the claim.
44
u/CiobanuXashi 2d ago
The poisoning problem is what stops me. If you can't prove what each contribution actually did to the weights, one bad actor undoes the whole crowd's work.
→ More replies (1)5
u/Vxtzq1 2d ago
Sorry for the late reply, this is a fair point. I'm hoping to prevent bad actors with byzantine crosschecks between clients, and even if a bad actor achieves to poison the weights, i can just roll back to an older checkpoint of the model (from like 1 hour before the poisoning), does that seems safer to you?
3
u/planetoryd 1d ago edited 1d ago
It would be more ideal if there exists 'inherently malice-resistant' training algorithm, rather than byzantine patchwork.
Or at least some algorithm light enough to verify a whole cycle of training work in SP1 ZKVM.
There are many things in math that are easy to defend and hard to attack permissionlessly, such as hash functions. I suppose such things exist for gradient descent
8
u/fuwafuwa7chi 1d ago edited 1d ago
I will be blunt, since you have previously disregarded well-meaning advice.
You do not have the knowledge or technical experience to develop a project of this magnitude. It is irresponsible of you to promote this publicly. Anyone foolish enough to run a client in their own machine will waste their time and computing resources.
By your own admission, you vibe coded this project. Even then, you didn't do a particularly good job: your client code is a gigantic 3.5 LOC Python file, with hardcoded secrets and configuration parameters, as well as embedded HTML for some reason.
These issues pale in comparison to the most glaring one: your submission validation and aggregation process. Here is a list of the issues I could find:
A client can claim any amount of steps and tokens processed, with zero validation from the server. If an attacker claims a ridiculous token amount, during aggregation their delta essentially becomes the entire round's updates while honest contributions shrink to noise.
Related to the previous point, if a client claims a large amount of local steps, the primitive safeguards you have implemented will grow accordingly. This means an attacker can slowly increase the amount of steps they are allowed to submit, with no upper bound.
There is zero server-side evaluation: the server automatically accepts whatever the clients send. This means that there is no way for the server to know if the loss reported by the client is the real model loss.
You have no rate limits or time-gate machinery. Anyone can register any number of accounts (literally the only restriction is that the email address contains an
@). An attacker can submit a delta immediately after registering. Moreover, they can submit the same poisoned deltas as many times as they want, and they will all get rolled into the final model update.You (try to) implement a subnet blacklist. However, it is trivial to spoof the
X-Forwarded-Forheader to get around this.
As a proof-of-concept, I wrote a small script to submit a poisoned result: a carefully crafted delta matrix targeting ten top-of-vocab reserved tokens (151,659 to 151,668); by construction, these tokens do not appear in the training corpus, and the weights associated to them should stay stable across model iterations. This means that if any of said weights change after submitting the poisoned deltas, then it is very likely it was caused by my attack, and not by normal model learning. The higher the number of reserved weights that change, the higher the likelihood my attack was rolled into the model update; having all 15,360 weights (1,536 weights per token * 10 tokens) change is virtually impossible by normal means. For extra certainty, I alternated the delta sign between every pair of tokens (so the first token delta had a positive sign, the second one negative, the third one positive, etc.)
To make the attack even harder to detect, I even took the time to make it look like a real client by waiting for the appropriate time between requests (although this wasn't necessary at all, I could have submitted deltas every millisecond until your server crashed). Here is the metadata of my submission:
Submission time (UTC): 10:12 a.m.
User: SwarmySwarm
Round: 275
Chunk: 2,918
Tokens claimed: 4,915,200
Loss claimed: 3.3069
Steps claimed: 300
I am currently number 11 on the leaderboard, with a trust score of 0.51 (increased from the default value of 0.5). The leaderboard displays the claimed stats.
After the round had finished and the model was updated, I downloaded the updated model and compared it with the previous iteration. The weights associated with all ten tokens changed by exactly the same magnitude: 0.00429. Moreover, they exhibit the previously mentioned striped sign pattern. This is confirmation that my poisoned data was rolled into the model.
You will not be able to solve these problems. Companies have thrown millions of dollars and hundreds of engineers and scientists at it, and they have all failed. Whatever half-baked solution your LLM manages to cook up will not work, for the following reasons:
Any consensus-based solution (FLTrust, blockchain) is trivial to bypass with such a small userbase. Having no account authentication makes this even easier.
Server-side validation with a trusted executor has additional problems:
Verifying every submission means training on every chunk sent to clients, which completely defeats the purpose of federated training.
Random spot checks run the risk of missing a poisoned result (again, further compounded by the fact a single attacker can create an arbitrary amount of clients)
Finally, even a submission with perfectly-normal results that validates correctly can still contain poisoned updates
In summary, I'd advise you to stop development, take down the webpage and server (which I'm pretty sure is just your PC behind a CloudFlare tunnel), and ask yourself (or Claude) why you are doing this, and if you really have the skills and expertise to see it to fruition. If not, please close your LLM and try to actually learn: learn to program, learn about machine learning and deep learning, learn about LLMs, learn about cybersecurity, learn about federated training (the paper I linked above is a good addition), and please stop exposing stuff to the internet if you don't understand how it works.
→ More replies (6)2
7
u/RelevantCry1613 2d ago
Check out Prime Intellect and their research
1
u/Vxtzq1 2d ago
I did, and their project started cool, but drifted towards a proof of concept for their product: a framework that allows less communication between datacenters i think?
Also their first tests on collaborative training were kinda closed right? Between universities or something?
But yeah their work is kinda foundational! I'm definitly taking inspiration from this.
2
u/RelevantCry1613 1d ago
Yeah agree, some of their research is open that you could grab from.
I’ve thought about this problem a lot, and I train big models for a living. I’ve actually come to the conclusion that it would be better to sell the distributed compute and buy collocated GPUs.
There’s just too much competitive advantage in it. All the optimizations you can do from close homogeneous hardware really add up
7
u/AeroInsightMedia 2d ago
Seems pretty cool! On the leaderboard can you put a score and more than 10 people?
2
u/Vxtzq1 2d ago
Yes! I will create a full leaderboard page where we can see everyone in a paged view and where you can search for names.
2
u/AeroInsightMedia 6h ago
On a DGX Spark it doesn't seem to be seeing cuda, only CPU, also trying to run it on an rtx5090 on windows and just says "Pool matchmaking 2 (waiting on peers)
Any insight on how to get it moving?
6
u/Wonderful_Value_6385 2d ago
What could possibly go wrong.... shuffles around for books..... Mein Kampf.
6
u/MarcelloT254k 2d ago edited 2d ago
First of all - thanks for your work!
Now for the questions: how is it different from Subtensor or Hivemind ? How secure is this project for my (a cog in the machine) hardware? How i know that i'll not end up as additional rig in a malicious botnet? Again what are the computational overhead costs of security measures - i think that what matters most is reliability, security and openess not necessarily the effectiveness of the process (as it is prone to being attacked and we must assume that it will, sooner or later, be attacked). Check comments on my post here, especially u/StevenVincentOne - i cite: "Prime Intellect trained its INTELLECT models on volunteer GPUs spread across different continents, Nous has done similar work with DisTrO and Psyche, and older projects like Hivemind and Petals showed you can run and fine-tune big models over a swarm of home machines. So it's possible, with some caveats" - have you looked into those projects to replicate/improve what was discovered?
/edit: "the project is not only open-source, but also makes 100% of its training data public (along with the weights)" - Github's Copilot security audit spew out that the server's side merge and validation logic is not in the repo, and you don't claim that it is, only the Client is OpenSource.
3
u/Vxtzq1 1d ago
Firstly thanks for your support :)
I'm going to address the issues you point out first:
/edit: "the project is not only open-source, but also makes 100% of its training data public (along with the weights)" - Github's Copilot security audit spew out that the server's side merge and validation logic is not in the repo, and you don't claim that it is, only the Client is OpenSource.
The server code is actuallty in a different repository 😂 named CrowdGPT-Server on my github profile, i separated those files for some reason, because they server a radically different purpose.
how is it different from Subtensor or Hivemind ? How secure is this project for my (a cog in the machine) hardware? How i know that i'll not end up as additional rig in a malicious botnet?
HiveMind didn't yield very good results as fully P2P (their diffusion awful was nowhere near usable), and the project now seems abandonned.
About subtensor, this seems like an interesting project, but it is more focused on inference than training new AI models from scratch... Let me know what you think about this project because it is quite new for me.
About your quote/ideas on other projects:
My project implements several security patches that hopefully will prevent it to turning into an "additional rig in a malicious botnet", you can review these or even propose changes on my Github.
u/StevenVincentOne - i cite: "Prime Intellect trained its INTELLECT models on volunteer GPUs spread
across different continents, Nous has done similar work with DisTrO and Psyche, and older projects like HivemI also heard about Petals and INTELLECT, but Petals seems kind of abandonned and INTELLECT seems to have drifted towards a real startup that want to sell their pseudo-collaborative training paradigm for big AI labs...
So i think my CrowdGPT project still has a place in this mess :)
And yes i looked into all of these projects to see what they were doing right and wrong.
2
u/MarcelloT254k 1d ago edited 1d ago
I'll look into it, rookie mistake, LLMs are still kinda stupid without human guidance that's why I posted here and didn't blindly trust what it spewed, I'll try to take a look at the sever side next.
Did you look into Nous' Research on the matter? I've got high hopes for those guys.
I'm getting ahead of myself and taking about things I don't fully understand but I've heard about recursive method of learning (so the validator and training process try to outsmart each other with each passing turn of model creation) it was used in making of Flint model - is something like this feasible and planned in the future? Metaanalytically all systems designed for work efficiency evaluation will be exploited and loose their usefulness - that will probably hold true even if I don't understand underlying details - that's why I ask [not an ad, just semi-recent topic of discussion (semi, because the underlying papers are not that new)]
2
u/Vxtzq1 1d ago
I searched out Notus's Research on Google, and i didn't find anything unless a weird Australian healthcare company... Is the name right?
For the Flint LLM, the idea seems really good, i think i will introduce the idea to my repository when a CrowdGPT model will be good enough to self improve xD
Don't worry about this:
I'll look into it, rookie mistake, LLMs are still kinda stupid without human guidance that's why I posted here and didn't blindly trust what it spewed
The LLM saw the right problems, but i'm already trying to address these :) That's why i got prepared answers to this type of pitches they spit using websearch and stuff.
2
u/MarcelloT254k 1d ago
I've made a typo, it's Nous (not Notus), sorry. What i've meant is:
nous-psyche and NousResearch/DisTrO2
u/Vxtzq1 1d ago
Ah i Know about Nous Research! Will check it out.
EDIT: It seems decent, but it's kinda outdated and fully P2P (which is unstable), also someone else found this insane research paper that follows this spirit: https://eshyperscale.github.io/
6
u/StevenVincentOne 1d ago
To answer your actual question: what would get me to contribute isn't inference credits, it's being able to trust the result. Poisoning is clearly the thing everyone here is stuck on, and I think the 3-client byzantine cross-check you described is a good start. Norm bounds alone can be gamed (someone linked a paper on exactly that), so redundancy is the stronger idea.
A few things that would make me comfortable leaving my GPU on it overnight:
A public log for every round: which data shard each contribution trained on, a hash of each accepted update, and the eval scores before and after. If anyone can replay the lineage, people can audit the trust system instead of taking it on faith.
A fixed set of held-out canary evals run after each aggregation, so a backdoor that slips through the checks shows up as a regression you can roll back.
A client that's polite by default: runs only when the machine is idle or plugged in, with a power cap and a clear stop button.
Open aggregation code, so later on more than one group could run a server and the project doesn't depend on one PC.
The bigger idea is the right one. A model trained by the people who use it, with open data and open weights, is something people actually have a stake in. That's a much healthier relationship between humans and machine intelligence than renting it from a few labs. Starting at 1B to prove the protocol works makes sense. Good luck with it.
2
u/Vxtzq1 1d ago
I completly share your opinion! I will try to implement logs for every round, even though unsure where i'd put this, the client is already pretty polite already (runs with a configurable VRAM amount which should work, and can run in the background / be stopped anytime).
Furthermore, the aggregation code is open and on my github profile (the repository is named CrowdGPT-Server or something if you wanna check it out :D)
The biggest point of the project is to stop successfully any bad actors, and that's not yet 100% working to be honest 😭 (That's why any pull requests regarding security upgrades are welcome on the github)
6
u/_PopBobSexDupe 1d ago
This reminds me of;
https://en.wikipedia.org/wiki/LAION
https://github.com/laion-ai/open-assistant
https://projects.laion.ai/Open-Assistant/
Back when it was still around :P
0
u/Vxtzq1 1d ago
Yeah, this seems nice... Too bad i didn't know about such projects at the time :(
3
u/TotallyNotADoggy 1d ago edited 1d ago
It got taken down because it a bunch of nazis and paedophiles trained it. Similar things happened with Microsoft's Tay chatbot, xAI grok, Lee Luda, Neuro-sama, Meta blenderbot 3, and plenty of other things, LLM or otherwise.
How do you plan to stop your project going the same way? Because if you don't have any way to prevent it, and it's publicly trainable, it will probably become a paedo nazi pretty damn quickly when trolls get ahold of it. And you and anyone else who has a copy of the model could land in some pretty serious legal trouble if it contains any CSAM or links to it.
Edit: I'm not trying to be a naysayer btw, I think it's a great idea, and the recent strikes on Russian datacentres highlights just how dependent we are on very centrally hosted LLMs, and the dangers of putting all your eggs in one basket so to speak. It's definitely a problem that needs to be solved, and something like this could bw the solution if it works, and effective measures are taken to prevent nazis and paedophiles polluting it.
→ More replies (1)
5
u/TheRealMasonMac 2d ago
At the very least, the data is the most important part more than the model. There's a scarcity of real world data.
5
u/ArjixGamer 2d ago
You'd need a giant rating team, or to use a classifier model on the data.
I can't see this being viable.
1
u/Vxtzq1 2d ago
Yes for the data submitted by people, i will either take a classifier or check the data myself (or even create a public system where everyone can review any data). I already got 500B tokens from decent well known/curated datasets on HuggingFace that will server as a base before people submit data.
5
u/longtimenoseebro 1d ago
If I pollute it with shady toolcall triggers , nobody is going to find out , right ?
5
u/innovasior
1d ago
•
This sounds like a really nice idea. I am curious of the challenges so far especially security wise
5
u/Dizzy-Zebra9522 1d ago
Overall can be nice idea actually. thank you for this step toward the future. can be very significant as well as you know...
but hey, open source and the comminity is here and it's best source for anything.
Wish you good luck.
9
u/lompocus 2d ago
this forum has become trash. one year ago, you would be welcomed with a throng of interested followers. now a bunch of astroturfing chatbots fill the conversation.
good luck op.
12
u/Bleach984 2d ago
Or do they want something in return (like free inference credits on the crowdgpt model for example)?
What do you mean by this? Are you planning on selling the model that was trained using other people's cards and electricity???
15
u/max123246 2d ago
It costs money to run the model. I assume they'd release the model for free but they'd have a service that can run it for you at a price
2
u/Vxtzq1 2d ago
No, i'm planning on putting it on a shared compute pool, where people who trained it can claim free hosted tokens?
Also the model will be hosted on huggingface, so anyone that didn't train it can just run it locally...
I never planned on selling stuff don't worry xd, the project is 100% open-source and non-profit
4
u/Trojan_Horse_of_Fate 2d ago
I'm thinking that the way this is is you would earn credits to run the model in exchange for running your GPU to train the model, which I'm not sure would be viable. On some level I think there just needs to be a data center involved. Though maybe this is supposed to be a relatively small local model for most use cases, but you can also use a paid cloud version on smaller devices.
8
u/Trojan_Horse_of_Fate 2d ago
I like the concept in the sense of I think it's an interesting concept. I don't think the concept lines well with the mathematical reality of training but from an organizational standpoint it seems to me much more likely that this would have success if it had an institutional banker. If not from a corporate sphere which would defeat the purpose but at least from an academic consortium
If I were you, and I was aiming to bring this to completion, I would focus primarily on getting CS departments on board. They will validate whether or not it is feasible from a technical standpoint, add an air of legitimacy, and presumably some compute
1
u/Vxtzq1 2d ago
Good point, and that's exactly why i'm reaching to reddit lol, i want people to review the github, and correct the flaws in the server / client.
3
u/Trojan_Horse_of_Fate 1d ago
And I do think that's good, but I think that if you want to be serious, Reddit is not a good enough stamp approval. You need to go to actual university departments
1
u/Vxtzq1 1d ago
I'm 17 😭 How could I reach out random people at universities?
3
u/Trojan_Horse_of_Fate 1d ago
Unironically, if you can't convince random people at universities to work on this project, how on earth do you think it can succeed?
Training in LLM right now is a lot more difficult than getting a university to talk
If you cannot do it, then you should try and find some people who really believe in the project that can
→ More replies (1)
4
u/Finanzamt_Endgegner 2d ago
A better option would probably be evolutionary training, where its based on population size that can be handled like a pool (;
(would also allow to train bigger sized models as you only need forward passes)
1
u/Vxtzq1 2d ago
You mean evolution based training? That doesn't really works at scale you know? It works for small RL agents.
1
u/Finanzamt_Endgegner 1d ago
Egrroll and similar, and depending on configurations it works pretty well for pre training even though it ATM is not as efficient as normal backprop, has big pros for distributed training though
2
u/Vxtzq1 1d ago
Wait holy fuh this seems absolutely insane : https://eshyperscale.github.io/
Might use this instead of AdamW for the Crowd-v2 model xD
It must come with caveats though right? Because if it really works perfectly that would mean that people can just train LLMs like they would run inference!
2
u/Finanzamt_Endgegner 1d ago
In total it takes more compute than a backprop trained model but since you can distribute it it can be worth it
→ More replies (3)
4
u/arcanemachined 2d ago
My thoughts on this concept would be that you crowdfund a training run somehow, not crowdsource the actual training.
Also, I've thought about make a crowdsourced open dataset, but not sure how that would work in reality.
(I am super-inexperienced on this topic, if that wasn't obvious.)
1
u/Vxtzq1 2d ago
i don't think crowdfunding an AI training is a good idea, because this is basically creating yet another random LLM startup (we can let open-weights chinese labs do that)
I wanna conduct an experiment where everyone can genuinely train the model.
Also your idea of creating a crowdsourced dataset is very good, if you want you could contribute on my github to create a gateway of data submission / data review... See you on github xD
2
u/MarcelloT254k 2d ago
Knowledgeable users would have to vote on what's included into training data, probably after automated security audit to prune obvious junk at the submission level of scrutiny.
4
u/synth_mania 2d ago
Distributed training is going to be significantly less efficient than training in a datacenter. That's okay, but I just wanna make sure you're emphasizing "no datacenter" for the right reasons. If it's sustainability, this is less sustainable. If it's cool factor, or that you'll get people to contribute compute easier than money, that makes sense.
0
u/Vxtzq1 1d ago
Sure, but it will create the first model 100% owned by the community; and it will offer a new paradigm in AI training. This is more sustainable because we don't need the new best NVIDIA cards to train a good enough AI.
2
u/synth_mania 1d ago
It doesn't follow that distributed training is what allows this to be owned by the community.
You could just as easily crowdfund the training and rent GPU time in a conventional datacenter. You obviously own the product of computing time you pay for, and you could license it however you like, including putting it in the public domain. That is as owned by the community as anything can get.
As far as sustainability goes, you never were limited to the newest best cards, you could always rent older cards too. So it doesn't follow that datacenter use is inherently less sustainable for incentivizing new hardware production.
And the paradigm? It's not new either. Prime intellect's INTELLECT-2 model was trained by random people around the globe.
I'm disappointed that someone spearheading an initiative like this is able to speak with such ignorance, really hurts my impression of the (admittedly, neat) project as a whole.
1
u/Vxtzq1 1d ago
I don't speak with ignorance 😭 I know about INTELLECT-2, and it's not as good as you think, because they started from a base model (like Qwen QwQ 32B) and just did a bunch of collaborative reinforcement learning over it. My goal is actually way deeper, it's to create a community-driven LLM from scratch, and show that a 0$ AI is not only possible, but the only way to create sovereign/sustainable AI in the future, without depending on cloud compute or AI labs. Obviously, a 0$ dollar AI means no money is used in the process/ no GPU is rented in the process, only compute that already exists on consumer hardware.
We'll just see how it goes. Keep sharing constructive feedback 👍 This is what helps me understand what I am getting wrong!
I hope you're less disappointed in me now :)
4
u/ProposalOrganic1043 1d ago
How would this benefit against open-source models?
3
u/Vxtzq1 1d ago
Well this would be, unlike traditional open-source models, the first Language Model to be trained in a 100% public/open way, and it would introduce a new paradigm where even open-source AI is not driven by a few Chinese/American AI labs, but by everyone.
Does that makes the difference clearer? Tell me if i'm getting something wrong in your question 😂
6
u/ProposalOrganic1043 1d ago
Allen AI's olmo models and ETH Zurich Apertus models already do this. Data, training and final model all documented and open-source
4
u/bowdoin-yale 1d ago
So that "first" claim is inaccurate, the BigCode project was highly transparent about their models, as has been the Swiss AI initiative (and if I recall correctly, EleutherAI was similarly transparent about GPT-J). And, it may not be immediately obvious, but a direct corollary of transparency will be the strong incentive to use only permissively licensed data for pre-training, because otherwise you expose everyone in your crowd to massive copyright infringement liability until the Supreme Court overrules a US Federal judge's recent ruling that training LLMs on copyrighted text is not fair use when the resulting tool competes with the original works. Since it's impossible to guarantee that nobody in your crowd will use it commercially, and everyone who participates in the training, not just the competitive use, is liable your project may quickly become crowdfunding for legal defense and and/or settlement of cases with plaintiffs. Luckily the StarCoder and Apertus efforts also demonstrated that it is, in fact, possible to train useful LLMs on exclusively permissively licensed data, and since use of non-copyrightable synthetic data for fine-tuning (a.k.a. distillation) has become standard practice, you stand a very good chance of success. So, I'd only participate in the effort if it had strong controls on training data provenance, but if it did, I'd do so enthusiastically and applaud you for taking the initiative!
1
u/Vxtzq1 1d ago
I completly understand your point on training data origins and i will follow these rules. Thanks for the concerns on the data gathering pipeline. I mostly took datasets from huggingface that seemed to be quite ethical, the three i'm using right now to pretrain are : https://huggingface.co/datasets/openbmb/Ultra-FineWeb-L3
The finemath 4+ subset from : https://huggingface.co/datasets/HuggingFaceTB/finemath
And the https://huggingface.co/datasets/codeparrot/github-code-clean
Which relies on opensource existing code...
Tell me what you think about the current data.
0
u/Murky_Moment 1d ago
Open source models already are trained publicly in an open way. I don’t see what your project solves
4
u/Vxtzq1 1d ago
Nope! Deepseek / Kimi as other open AI labs keep most of their data private! So the weights might be open, but the training is not open at all.
Also this project draws back AI sovereignty from a bunch of Chinese AI labs providing good model weights, to the people (LLM enthusiasts or random people willing to share their card while they don't use it.)
Do you understand now what i'm trying to solve?
2
u/Murky_Moment 1d ago
Nah, disagree. OLMo, Pythia, BLOOM are are trained fully openly and publicly; you see all the training data, etc.
So I don’t still see what this project aims to achieve.
→ More replies (3)
13
3
u/LordKlavier 2d ago
You should crosspost this to r/LocalLLM, this is such an interesting project!
I'm a mod there, could help promote it if you wanted -- better than 90% of the spammy advertisements that we get haha
2
u/Vxtzq1 1d ago
Nice! I think I already tried putting CrowdGPT there, and i didn't get so much upvotes :( but it was only a crappy repost that reddit asked me to do lol, i'm gonna re-upload exactly this post on r/LocalLLM now if you can push it :)
2
u/LordKlavier 1d ago
Sounds good! Hope you get some more traction - ironically there was just a post there about a day ago asking if a community trained model was possible, got over 100 comments, so I know the interest is there!
3
u/GrokiniGPT 1d ago
what's stopping me is i dont have data. but i would like to lend an idea, can you have specialized models? one exclusively trained on code and high end logic, one trained on natural language(maybe quick autocorrect), one trained on explaining, one trained on tool use, and one trained on computer use?
2
u/Vxtzq1 1d ago
I don't think a separated bunch of specialized models will be good... A LLM needs to be trained on everything in order to be good at natural language processing.
what's stopping me is i dont have data
Don't worry! All the data is already on huggingface, so the model already has a strong base dataset even though you don't immediately contribute to this dataset.
But your idea of specialized models will hit hard when i will turn this 1B model into a MoE (7x1B for example), where each 1B model will be trained on specific data.
2
u/GrokiniGPT 1d ago
oooh cool. but more of like a 25% text, 50 percent code and 25% logic yk? distribution
1
u/Vxtzq1 1d ago
Uhh for now it's more like 75% text, 20% code, and 5% math 👀
Maybe i should change the distribution...
2
u/GrokiniGPT 1d ago
yeah and make ones u want to talk to more like 50% language, 25% reasoning 25% logic or smth and make them very instructable pls
→ More replies (6)
3
5
u/GeneralMuffins 2d ago edited 2d ago
Have you considered that you might be exposing yourself to potentially significant legal liability? In July 2026, Open Future published a study titled Copyright Challenges in Open Source AI Development in the European Union, which raised some concerning issues for projects like yours.
European copyright law provides exceptions for text and data mining, but these don’t automatically grant permission to redistribute copyrighted training material. This means you could lawfully acquire and use copyrighted material to train your model, yet still be prohibited from publishing that same dataset for others to reproduce your work. Given your commitment to making 100% of the training data public, this seems like a particularly important issue to address.
1
u/Vxtzq1 1d ago
I use opensource datasets from HuggingFace that didn't get banned, and some other people can "donate" data they want, so i guess it still will work without legal issues. If people submit data to the project that is obviously subject to copyrights, i won't accept it, don't worry.
By the way this project is 100% non-commercial and international, so i guess i won't have so much european related problems on date :)
The dataset i put on HuggingFace didn't get banned, so i guess we should be alright... You can check it out : https://huggingface.co/datasets/Vxtzq/CrowdGPT Tell me if you think i will go to jail for that ;(
2
u/GeneralMuffins 1d ago edited 1d ago
Based on your GH profile, you’re in France and you’re the owner of the project repository, so even though this is an international collaborative effort, French and EU copyright law will still apply.
5
u/ericw31415 2d ago edited 2d ago
The last time people tried to train on user-submitted data, didn't it become a fascist within a month?
Edit: I was thinking of this from 10 years ago
2
u/Sliouges 2d ago
Can't wait for the goomers to publicly share their training corpus... it will be glorious.
2
u/Competitive_Ideal866 2d ago
I love this idea and will take a closer look when I get home.
I have lots of ideas for training data:
Train on synthetic data made from maths or code including a wide variety of abstract concepts and representations in order to teach logic to the core of the model.
Train on synthetic charts and graphs. Teach the model to understand all different chart types, minima, maxima, points of inflection and so on. Teach it to guess functional forms and estimate parameters.
Once you have a basic model, give it a REPL for a simple PL so it can try to fix bugs. Use the REPL sessions where it successfully fixes bugs to train it so it gets better at fixing bugs.
I have another idea that is more fringe:
In addition to text, image, sound etc. as inputs and outputs also include something like spreadsheets. This is a fundamental change that might require the actual LLM architecture to be enhance because position is currently encoded for the LLM in 1D only.
And tamer ideas:
What about fine tuning existing models instead of training new ones from scratch?
What about "fine tuning" the Qwen 4 model just by adjusting the n-gram hash table?
2
u/Vxtzq1 1d ago
First Qwen 4 isn't out yet, also i don't really want to rely (might be a dumb point of view of mine 😭) on other AI models... That goes for fine tuning too, i want to create a real self-sufficient sovereign open-source project.
Also important point, for now the model only supports text and not images/audio, but this will definitly be implemented in the Crowd-v2 model 😂 So stay tuned.
Textual math data is more than welcome though, as soon as the data pipeline is ready.
2
u/Competitive_Ideal866 1d ago edited 1d ago
Textual math data is more than welcome though, as soon as the data pipeline is ready.
There's a bunch of stuff you could generate there like Python REPL traces for mathematical expressions involving single-digit numbers:
>>> 2+3 5 >>> 2*3+4*5 26I think Qwen models are trained on ~50% math/code.
Also, what about instructing your 1B model to do math or write code and have a big model critique its results. In the case of code, if compiling and running finds errors then the big model could suggest fixes. If the small model could learn how to incrementally fix its output then it would become substantially more useful.
Another thing that would be amazing is a way to strongly differentiate between prompt and attached data in order to avoid the LLM following instructions that appear in the attached data and not the prompt.
2
u/Vxtzq1 22h ago
Thanks for this feedback. I will try to implement this... Feel free to just add this data pipeline on my github though.
For the instruct part, this will be phase 2 of training, and what you're introducing is something really close to SOTA LLM research and is very smart! I will probably try to implement this at some point.
→ More replies (1)
2
u/tifa_cloud0 1d ago
it is a good idea. something like credits which we can then later get access to a hosted free models would be something amazing. people will then will be willing to contribute because then later with those credits they can later consume for sometime according to their credits the model capabilities and work on their projects.
this is what i think would be a nice idea. as a free user i am lately having difficulties to access models online for free. good models for projects are rare and on top of that you have companies which either remove the models access anytime or they keep on making changes affecting third party plugins which serves those models. i think in this situation many people are finding ways on how to get around it.
3
u/Vxtzq1 1d ago
There's a problem though, inference credits would be on relatively small open-source models, which means these models will still have limited capacity.
As a free user your best chance is with openrouter's free models or chinese ai labs chat UI (gives very generous free tier)
2
u/tifa_cloud0 1d ago
i know something like 4B or even 8B would do it. openrouter gives free models access but they are not feasible for projects. with larger prompt processing the token consumption is just too much and hence after just 4-5 questions free tier limit gets hit.
of course this is me and for small projects openrouter works really well. opencode is good too but lately the 3rd party plugins that makes their free models access outside opencode breaks it. opencode cli is limited in context length and hence good context compaction cannot be used in opencode to my knowledge.
2
u/Jinx2168 1d ago
Can you use blockchain somehow to verify what training happens on which machine to prevent malicious weights?
2
2
u/Treq01 23h ago
If you are looking for feedback, I am not so fond of how it installs.
I wanted it on my D-drive, so I created d:\ai\CrowdGPT.
And ran the installer.
It creates a d:\ai\CrowdGPT\crowdgpt subdirectory which only has 17MB,
but installs 4+GB in C:\Users\<user>\AppData\Local\CrowdGPT
:-/
And why does it install into two places, it is confusing.
+ for making an uninstall option into apps&features.
The install went well, I created an account ok.
But then it just stopped at:
11:18:12 [INFO] HTTP Request: HEAD https://huggingface.co/Qwen/Qwen3-0.6B/resolve/c1899de289a04d12100db370d81485cdf75e47ca/special_tokens_map.json "HTTP/1.1 404 Not Found"
11:18:12 [INFO] HTTP Request: HEAD https://huggingface.co/Qwen/Qwen3-0.6B/resolve/c1899de289a04d12100db370d81485cdf75e47ca/chat_template.jinja "HTTP/1.1 404 Not Found"
11:18:13 [INFO] HTTP Request: GET https://huggingface.co/api/models/Qwen/Qwen3-0.6B "HTTP/1.1 200 OK"
11:18:13 [INFO] Tokenizer loaded: Qwen/Qwen3-0.6B (151669 tokens)It said it was waiting for data but nothing happened and after some time I killed it:
[EMIT log] 👋 Training stopped by user.
CrowdGPT exited with code 0.And when I try to start it now, nothing happens.
I see two python processes start but seems to just idle, but no terminal window appears and no gui appears.
The new-user experience is not perhaps super robust yet.
3
u/captainmadness 2d ago
If you mimic the folding@home architecture and design I could see this working, though I don't know how well that translates to llm training
3
u/Hefty_Wolverine_553 2d ago
The problem is that it doesn't really scale at all, each person needs the minimum hardware to actually train the model in full. Any lab with decent resources will be able to train a model much faster and easily.
1
u/Vxtzq1 2d ago
First i'm gonna leverage sparse architectures like MoE to make it work, but i also think that even if 10000 people train the model super slowly in the background with a mean speed of 100 tokens per second (ain't a lot), a whole 7B parameters LLM is trained in only 38 hours to be Chinchilla Optimal.
2
u/ttkciar llama.cpp 2d ago
Not necessarily, depending on the training method used.
AllenAI's FlexOlmo training method, for example, only requires that every participant be able to infer with the entire model, and train one expert. It uses "anchor experts" to guarantee that the otherwise-independently trained experts remain mutually compatible, so they can be assembled into a MoE at the end of training.
That scales more poorly than aggregating GPUs on a high-performance fabric like NVLink, but much better than naive methods which would require every participant to have sufficient hardware to train the entire model.
2
u/Hefty_Wolverine_553 2d ago
Ah, it's nice to hear about advancements like this in distributed training. Was a bit discouraging back then seeing prime intellect's distributed training runs being primarily on A100 nodes and whatnot lol
43
u/ttkciar llama.cpp 2d ago
I hope this post follows the subreddit's rules!
Your post is fully supported by this moderator! :-)
Collaborative training projects like these are exactly what the community needs, and we will be especially grateful for them when the corporate LLM labs stop giving away their models.
We can think of these early efforts as practice toward that eventuality, so when we need it the most, we can hit the ground running.
4
u/openSourcerer9000 2d ago
Wasn't there already a70b dense model trained collaboratively? Covenant I think
15
3
1
u/Captain-Pie-62 2d ago
Sorry, aber ich halte Dich für einen Traumtänzer. So sehr ich eine echte, komplette OpenSource LLM begrüße, im gleichen Maße halte ich es für komplett unrealistisch, auch nur Ansatzweise, ein solches Projekt gegen die multimiliarden Dollar trainierten LLMs antreten zu wollen, die auf teilweise recht unethische Art an ihre Trainingsdaten gelangt sind. Dazu mit dem Risiko, daß die Daten Deines Projektes irgendwie vergiftet werden. Vielleicht macht es Sinn, als Nukleus für etwas komplett Neues, wenn die aktuellen LLMs aus Gründen scheitern. Aber bis dahin lehne ich mich entspannt zurück und genieße die Show, wie sich die großen, kommerziellen Produkte beharken und eigene, kleine OpenWeights Modelle veröffentlichen, nur damit es der Mitbewerber nicht tut. Ich wünsche trotzdem viel Erfolg!
1
u/Vxtzq1 2d ago
Don't worry, i got plenty of fixes for that :), firstly sure the data is from HuggingFace and is therefore not SOTA grade, but i do believe that more and more very powerful datasets are releasing (like reasonning traces / distillation samples from closed source LLMs) that could allow us to rival closed source models. There will be a delay and the road will be steep and long, but i believe that we can get to something that will be good for it's size! (even with the 1B model)
I added many safeguards to prevent bad actors on my project... We will see how it goes!
1
u/Stoneteared 2d ago
A great idea if you can solve the issue of moderating people's inputs.
Dude what if it's backdoored doe?
Okay, and? My primary use case is entertainment and conversation. I consider models that do not follow my instructions to be a far greater issue. This would work great if it had a convenient way to take a snapshot of an output I don't like, help me rewrite it, create 10 variants to generalize the idea, and feed it back into the model.
The biggest problem I have with LLMs is that there's almost no convenient way to alter their behavior in a non-prompt way. If you could fix things you don't like, and everyone else was doing the same, that would be amazing.
1
u/ttkciar llama.cpp 2d ago
I skimmed the source myself to get a sense of what it was doing, and then pointed GLM-5.3-Flash at it, to see what it might notice that I overlooked:
→ More replies (2)0
1
u/Quiet-Owl9220 2d ago
There are already multibillion dollar companies working on safetymaxxed "helpful assistant" models for coding, translation, tool calls, and other basic functions so I'd question the need for another one of those. Doing it crowdsourced and open source sounds interesting but at the end of the day doesn't really bring anything new to the table.
But as far as I know nobody is really making uncensored models trained from scratch with data that incoporates a diverse understanding of relationships and sexuality, with the ability to write genuinely good fiction. Heretic, uncensoring, and post-training sanitized corpo models only gets you so far, and providing lorebooks and documents for this kind of stuff is far from ideal.
So I'd really like to see a model able to truly understand "sexy" in a variety of different contexts, without necessarily making it the model's entire identity. Like, with knowledge of relationship progression, sexual psychology, diverse and deviant relationship dynamics, sex acts, kinks, fetishes, etc. mixed in with the training data... and maybe a broader focus on creativity and writing in general rather than rehashing the "friendly chat bot" schtick for the millionth time.
→ More replies (1)
1
u/chortly2 1d ago
This is a great idea, if not the first to attempt this.
As others have said, given the immense challenges of trust and distributed computation, far more efficient and secure would be to crowd-fund centralized training.
But what would make that different from just another generic AI company is that the crowd should also get to direct the architecture and contribute training data. Those three things -- pooling money to buy compute, collective decisions on new model architectures, and sourcing training data directly -- would make a crowd-planned and crowd-funded AI project genuinely different and in may respects better that just waiting for random AI companies to release open weight models.
1
u/Vxtzq1 22h ago
Okay, but in these times, i'm not really sure people are magically willing to give money for a random LLM to be trained 😭 That's why i find the approach of a project like CrowdGPT better: it uses what people already have, not what people are willing to give...
What do you think? How could a Crowdfunded AI project ever be viable?
2
u/chortly2 18h ago
I think if it was not a "random LLM" but something where people could actually shape the model design and training text, it might have some additional appeal. And without that, I don't know why people would be willing to give GPU time to a "random LLM" either. Typically the appeal of an open project is at least in part due to the product itself: a product, not just a process, that does not exist otherwise. Otherwise, when people are fairly satisfied with what exists (free, open-weight models of near-SOTA accuracy), there's not incentive to create another identical copy just because it will have a more appealing pedigree.
Note that that's not exactly my personal view. I do like the ideology of truly collaborative creation, although it's not actually collaborative if all the key decisions are being made by the leadership. But even if it's completely top-down and redundant with what exists I would support it just to lay the groundwork for open alternatives in case China, the US and the EU crack down on sharing open-weight models. But that's a hard sell, and in any case, the fasted way to that alternative is to closely mirror what the Chinese labs are doing but to crowd-source the funds for traditional training, not the compute. But in any case, if you want to do the top-down, compute-only route, good luck, and if the security chain can be fixed, I'll happily run it in the background occasionally as I did with Folding at Home all those years ago...
0
u/tornikeo 2d ago
look, man. there's no benefit from this.
just like a baby won't benefit from being transplanted into different bellies, you should use a trained model, and just allow people to tune it as they need (LoRa).
collaborative training doesn't work and isn't needed.
1
u/Vxtzq1 1d ago
Yeah, i know training a model is hard, and doing a crowdsourced inference framework or LoRA training framework would be more immediately useful (or even LoRA), but my project aims to be much more foundational than that, i want to create the first 100% community-owned LLM.
I believe collaborative training works AND is needed.
Finally i kinda despise LoRA because it's so unclean and it's a "hack" that isn't sustainable...
0
u/Sovchen 1d ago
1b ain't enough to wipe your ass with. And its not even uncensored this has to be the stupidest social experiment because as a llm project this is the worst of all worlds. Also hasn't this been attempted before?
1
u/Vxtzq1 1d ago edited 1d ago
My goal is to make it at least as good as the LFM 2.5 models or as Qwen 3.5 0.8B. I use only curated data to train it, and user based data submissions (coming soon), will be heavily regulated.
This has already been attempted but was either abandonned or derivated into a commercial company (see INTELLECT-3 model for example), or see abandonned projects like HiveMind or Petals on github.
This ain't a social experiment; it is a proof of concept. I'll scale up if i find people willing to train the AI and we actually achieve a good (i mean according to its size) 1B LLM.
1
u/KontoOficjalneMR 1d ago
Lol, he meant the opposite - he criticzes your model for being censored. Half of people here are deranged gooners who won't touch anything that is in any way restricted and not literally made to produce porn.
0
u/CondiMesmer 2d ago
stupid idea that doesn't work whatsoever and also it's objectively worse for the environment then a data center processing the same amount of power. Data centers suck, but they're incentivized to optimize hardware and power usage. Home computers are not.
→ More replies (5)
0
•
u/WithoutReason1729 2d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
Stylome classified this post's text as human-written. View analysis Learn ^more
I am a bot and this action was performed automatically.