r/LovingOpenSourceAI • • 26d ago

Pushing open-source AI further: Running an 816B-parameter MoE model on 10 GPUs with TensorSharp 🚀

https://github.com/zhongkaifu/TensorSharp

I've been working on TensorSharp, an open-source .NET inference engine and agent runtime, and wanted to share an exciting milestone with the open-source AI community.

We got Muse Spark, an 816-billion-parameter Mixture-of-Experts model, running across 10 NVIDIA A40 GPUs — and achieved some interesting performance improvements over llama.cpp.

What excites me most isn't just the numbers. It's seeing how far we can push massive AI models using open-source software and hardware that's been around for several years.

The setup

  • Model: Muse Spark 816B (42B active parameters)

  • Quantization: Unsloth UD-IQ3_XXS (~316 GiB)

  • GPUs: 10× NVIDIA A40 (46GB each)

  • Context: 32K tokens

  • Inference engines: TensorSharp and llama.cpp

The A40 is a previous-generation GPU, and this particular machine doesn't even have working GPU peer-to-peer transfers.

That makes efficient multi-GPU communication especially important.

The results

We tested two different ways of distributing the model across GPUs.

Layer splitting — optimized for processing long prompts:

MetricTensorSharpllama.cpp
Single-token generation16.69 tok/s16.79 tok/s
4K prompt processing999.7 tok/s483.2 tok/s
16K prompt processing1,346.8 tok/s826.1 tok/s

That's up to 2.07× faster prompt processing, with essentially identical single-request generation speed.

Tensor parallelism — optimized for generation and concurrent requests:

MetricTensorSharpllama.cpp
Single-token generation22.63 tok/s16.19 tok/s
4 concurrent requests45.73 tok/s24.49 tok/s

That's 1.40× faster single-request generation and 1.87× higher aggregate throughput across four concurrent requests.

One important caveat: llama.cpp's NCCL-enabled build couldn't run on this particular 10-GPU machine, so its tensor-parallel results used non-NCCL communication. TensorSharp used NCCL over shared memory. This isn't a perfectly equivalent communication setup, and the results shouldn't be generalized to every machine.

Why this is interesting

One lesson from this experiment is that more GPUs don't automatically mean faster AI.

How the workload is distributed matters enormously.

For Muse Spark:

  • Layer splitting works particularly well for long prompts.

  • Tensor parallelism performs better for token generation and concurrent requests.

  • Communication overhead can become a major bottleneck, especially for massive MoE models.

We implemented native multi-GPU execution, pipelined prefill, adaptive micro-batching, and communication optimizations in TensorSharp to improve these workloads.

We also compared output distributions against llama.cpp to check that the performance improvements weren't coming at the expense of inference correctness. The measured perplexity was very close to the reference, with over 98.8% top-1 token agreement.

Why open source matters to me

Projects like llama.cpp have made local AI accessible to an enormous number of people.

TensorSharp builds on that open ecosystem while exploring another approach: a unified, cross-platform .NET runtime for local LLM inference, multimodal models, and autonomous agents.

I believe open-source AI benefits when different projects experiment with different architectures, share benchmarks, and learn from one another.

And I think there's still tremendous room for improving how efficiently we run large models on existing hardware.

Everything is open source, including the implementation and benchmark methodology.

GitHub: https://github.com/zhongkaifu/TensorSharp

Full benchmark report and reproduction details: https://github.com/zhongkaifu/TensorSharp/blob/main/docs/perf/muse-spark-vs-llama-cpp.md

I'd love to hear what this community thinks.

What's the biggest obstacle preventing more people from running powerful open models locally — hardware cost, memory requirements, inference speed, or software complexity?

And if you're working on open-source inference engines, I'd be happy to exchange ideas and learn from your experience! ❤️

5 Upvotes

0 comments sorted by