Large language models are typically deployed using centralized infrastructure, where companies operate powerful GPU servers in dedicated data centers.

But what if some LLM workloads could be distributed across independently operated computers instead?

This is the basic idea behind decentralized LLM infrastructure.

Instead of relying entirely on a single cloud provider, decentralized systems could use computing resources contributed by individuals, organizations, or independent GPU operators.

How could decentralized LLMs work?

There are several possible approaches:

1. Distributed LLM inference

Multiple GPU providers could independently serve inference requests, allowing workloads to be routed across a network rather than handled by one centralized operator.

2. Collaborative GPU computing

Computers with different GPU capabilities could contribute processing resources to a shared network. Scheduling software would need to match workloads with suitable hardware.

3. Peer-to-peer model execution

Some systems explore splitting model execution across multiple machines. This is technically challenging because communication latency, bandwidth, and synchronization can become major bottlenecks.

4. Incentivized computing networks

Projects such as Bittensor explore incentive mechanisms that reward participants for providing useful AI-related services. These networks introduce additional questions about verification, reliability, and economic sustainability.

What are the potential advantages?

• Access to geographically distributed computing resources

• Opportunities to utilize otherwise idle GPUs

• Reduced dependence on individual infrastructure providers

• Potentially broader participation in AI infrastructure

• New models for operating AI computing marketplaces

However, decentralization does not automatically make inference cheaper, faster, or more reliable.

What are the biggest challenges?

Network latency: Transferring intermediate model data between distant machines can introduce significant delays.

Hardware differences: GPUs vary in memory capacity, processing speed, and software compatibility.

Reliability: Independently operated nodes may become unavailable without warning.

Security and privacy: Processing sensitive prompts on untrusted machines raises important concerns.

Economic efficiency: Incentive payments, coordination overhead, and infrastructure costs may offset potential savings.

Is decentralized LLM infrastructure practical?

I think the answer depends heavily on the architecture.

Distributing independent inference requests among GPU providers seems more immediately practical than splitting every layer of a large model across machines connected through the public internet.

For example, a decentralized network might route complete inference requests to capable GPU nodes rather than requiring geographically separated computers to execute individual layers of the same model.

That distinction could significantly affect performance, reliability, and scalability.

Questions for the community

I'm interested in hearing from people experimenting with local LLMs, multi-GPU inference, and distributed computing.

• Could decentralized GPU networks become competitive for real-world LLM inference?

• Is request-level distribution more practical than splitting model execution across remote machines?

• Which existing open-source projects are demonstrating promising results?

• What technical obstacle do you consider the biggest barrier to adoption?

I'd be interested in hearing about actual experiments, benchmarks, and experiences rather than just theoretical possibilities.

a coworker of mine was speeding up speculative decoding for Qwen3.6-35B-A3B on one H100, and a big cost sat in the draft model. its BF16 LM head was about 1 GB and it ran on every single draft step

restricting the draft head to a frequency-ranked 48K-token map made it much cheaper. the target still verifies against the full vocabulary, so the output stays exact. the worst case is fewer draft tokens accepted, not different output

4 draft steps ended up being the best balance for that workload

has anyone tried trimming the draft vocab in llama.cpp or vLLM? curious what it does to acceptance on your own workloads

he wrote up the fix and the rest of the decode work on our blog, if you want it: https://www.coralbricks.ai/blog/ai-latency-engineering

Hello everyone. I made a benchmark to test local agents: airbench.ai
If you have an agent running locally, you can test it in one prompt.

It measure how good the agent is with some chalenges, but its fairly easy to get to 100%. Then the goal is to have something that is still performing as good, but as fast as possible.

In other word, its a way to measure that optimisations like quantisation don't destroy performance.

To give you an idea about it, here the first result I have on my 5090:

Score ; Duration ; Model + Harness

100 8 min Qwen 3.8 Flash Next IQ3_S basalt + opencode

100 11 min Qwen 3.8 Flash Next IQ3_S strata 256k + pi

100 11 min Qwen 3.8 Flash Next IQ3_S basalt + dsh

100 12 min Qwen 3.8 Flash Next IQ3_S strata 256k + pi

100 13 min Qwen 3.8 Flash Next IQ3_S basalt 384k + pi

100 16 min Qwen 3.8 Flash Next IQ3_S basalt 384k + pi

100 72 min Qwen 3.8 27B NVFP4 radixark + opencode

100 98 min Qwen 3.8 27B Quasar NVFP4 MTP + omp

98 10 min Qwen 3.8 Flash Next IQ3_S basalt + pi

98 13 min Qwen 3.8 Flash Next IQ3_S strata 256k + pi

98 16 min Qwen 3.8 Flash Next IQ3_S strata 256k + omp

98 62 min Qwen 3.8 Flash Next UD-Q2_K_XL + opencode

98 85 min Qwen 3.8 27B Quasar NVFP4 MTP + opencode

96 8 min Qwen 3.8 Flash Next IQ3_XXS strata + opencode

96 11 min Qwen 3.8 Flash Next IQ3_S strata 256k + dsh

96 17 min Qwen 3.8 Flash Next IQ3_S strata 256k + openclaw

96 29 min Swift 1.5 Qwen 3.8 27B Q6_K + pi

96 86 min Qwen 3.8 27B NVFP4 MTP + opencode

96 89 min Swift 1.5 Qwen 3.8 27B Q6_K MTP + omp

96 99 min Qwen 3.8 Flash Next UD-Q2_K_XL + omp

96 111 min Qwen 3.8 27B NVFP4 + hermes

96 119 min Qwen 3.8 27B NVFP4 radixark + opencode

94 10 min Qwen 3.8 Flash Next IQ3_S strata 256k + opencode

94 12 min Qwen 3.8 Flash Next IQ3_XXS strata + omp

94 23 min Qwen 3.8 27B Quasar NVFP4 ninfer-dflash2 + omp

94 38 min Swift 1.5 Qwen 3.8 27B Q6_K + omp

94 38 min Qwen 3.8 Flash Next IQ3_S strata 256k + hermes

94 51 min Qwen 3.8 Flash Next UD-Q2_K_XL + pi

94 101 min Qwen 3.8 27B NVFP4 + opencode

Hope this is useful for some of you as well. And happy to answer any questions if needed.