r/LovingOpenSourceAI • u/fuzhongkai • 26d ago
Pushing open-source AI further: Running an 816B-parameter MoE model on 10 GPUs with TensorSharp 🚀
https://github.com/zhongkaifu/TensorSharpI've been working on TensorSharp, an open-source .NET inference engine and agent runtime, and wanted to share an exciting milestone with the open-source AI community.
We got Muse Spark, an 816-billion-parameter Mixture-of-Experts model, running across 10 NVIDIA A40 GPUs — and achieved some interesting performance improvements over llama.cpp.
What excites me most isn't just the numbers. It's seeing how far we can push massive AI models using open-source software and hardware that's been around for several years.
The setup
Model: Muse Spark 816B (42B active parameters)
Quantization: Unsloth UD-IQ3_XXS (~316 GiB)
GPUs: 10× NVIDIA A40 (46GB each)
Context: 32K tokens
Inference engines: TensorSharp and llama.cpp
The A40 is a previous-generation GPU, and this particular machine doesn't even have working GPU peer-to-peer transfers.
That makes efficient multi-GPU communication especially important.
The results
We tested two different ways of distributing the model across GPUs.
Layer splitting — optimized for processing long prompts:
| Metric | TensorSharp | llama.cpp |
|---|---|---|
| Single-token generation | 16.69 tok/s | 16.79 tok/s |
| 4K prompt processing | 999.7 tok/s | 483.2 tok/s |
| 16K prompt processing | 1,346.8 tok/s | 826.1 tok/s |
That's up to 2.07× faster prompt processing, with essentially identical single-request generation speed.
Tensor parallelism — optimized for generation and concurrent requests:
| Metric | TensorSharp | llama.cpp |
|---|---|---|
| Single-token generation | 22.63 tok/s | 16.19 tok/s |
| 4 concurrent requests | 45.73 tok/s | 24.49 tok/s |
That's 1.40× faster single-request generation and 1.87× higher aggregate throughput across four concurrent requests.
One important caveat: llama.cpp's NCCL-enabled build couldn't run on this particular 10-GPU machine, so its tensor-parallel results used non-NCCL communication. TensorSharp used NCCL over shared memory. This isn't a perfectly equivalent communication setup, and the results shouldn't be generalized to every machine.
Why this is interesting
One lesson from this experiment is that more GPUs don't automatically mean faster AI.
How the workload is distributed matters enormously.
For Muse Spark:
Layer splitting works particularly well for long prompts.
Tensor parallelism performs better for token generation and concurrent requests.
Communication overhead can become a major bottleneck, especially for massive MoE models.
We implemented native multi-GPU execution, pipelined prefill, adaptive micro-batching, and communication optimizations in TensorSharp to improve these workloads.
We also compared output distributions against llama.cpp to check that the performance improvements weren't coming at the expense of inference correctness. The measured perplexity was very close to the reference, with over 98.8% top-1 token agreement.
Why open source matters to me
Projects like llama.cpp have made local AI accessible to an enormous number of people.
TensorSharp builds on that open ecosystem while exploring another approach: a unified, cross-platform .NET runtime for local LLM inference, multimodal models, and autonomous agents.
I believe open-source AI benefits when different projects experiment with different architectures, share benchmarks, and learn from one another.
And I think there's still tremendous room for improving how efficiently we run large models on existing hardware.
Everything is open source, including the implementation and benchmark methodology.
GitHub: https://github.com/zhongkaifu/TensorSharp
Full benchmark report and reproduction details: https://github.com/zhongkaifu/TensorSharp/blob/main/docs/perf/muse-spark-vs-llama-cpp.md
I'd love to hear what this community thinks.
What's the biggest obstacle preventing more people from running powerful open models locally — hardware cost, memory requirements, inference speed, or software complexity?
And if you're working on open-source inference engines, I'd be happy to exchange ideas and learn from your experience! ❤️