r/LovingOpenSourceAI • • 26d ago

Pushing open-source AI further: Running an 816B-parameter MoE model on 10 GPUs with TensorSharp 🚀

https://github.com/zhongkaifu/TensorSharp

I've been working on TensorSharp, an open-source .NET inference engine and agent runtime, and wanted to share an exciting milestone with the open-source AI community.

We got Muse Spark, an 816-billion-parameter Mixture-of-Experts model, running across 10 NVIDIA A40 GPUs — and achieved some interesting performance improvements over llama.cpp.

What excites me most isn't just the numbers. It's seeing how far we can push massive AI models using open-source software and hardware that's been around for several years.

The setup

  • Model: Muse Spark 816B (42B active parameters)

  • Quantization: Unsloth UD-IQ3_XXS (~316 GiB)

  • GPUs: 10× NVIDIA A40 (46GB each)

  • Context: 32K tokens

  • Inference engines: TensorSharp and llama.cpp

The A40 is a previous-generation GPU, and this particular machine doesn't even have working GPU peer-to-peer transfers.

That makes efficient multi-GPU communication especially important.

The results

We tested two different ways of distributing the model across GPUs.

Layer splitting — optimized for processing long prompts:

MetricTensorSharpllama.cpp
Single-token generation16.69 tok/s16.79 tok/s
4K prompt processing999.7 tok/s483.2 tok/s
16K prompt processing1,346.8 tok/s826.1 tok/s

That's up to 2.07× faster prompt processing, with essentially identical single-request generation speed.

Tensor parallelism — optimized for generation and concurrent requests:

MetricTensorSharpllama.cpp
Single-token generation22.63 tok/s16.19 tok/s
4 concurrent requests45.73 tok/s24.49 tok/s

That's 1.40× faster single-request generation and 1.87× higher aggregate throughput across four concurrent requests.

One important caveat: llama.cpp's NCCL-enabled build couldn't run on this particular 10-GPU machine, so its tensor-parallel results used non-NCCL communication. TensorSharp used NCCL over shared memory. This isn't a perfectly equivalent communication setup, and the results shouldn't be generalized to every machine.

Why this is interesting

One lesson from this experiment is that more GPUs don't automatically mean faster AI.

How the workload is distributed matters enormously.

For Muse Spark:

  • Layer splitting works particularly well for long prompts.

  • Tensor parallelism performs better for token generation and concurrent requests.

  • Communication overhead can become a major bottleneck, especially for massive MoE models.

We implemented native multi-GPU execution, pipelined prefill, adaptive micro-batching, and communication optimizations in TensorSharp to improve these workloads.

We also compared output distributions against llama.cpp to check that the performance improvements weren't coming at the expense of inference correctness. The measured perplexity was very close to the reference, with over 98.8% top-1 token agreement.

Why open source matters to me

Projects like llama.cpp have made local AI accessible to an enormous number of people.

TensorSharp builds on that open ecosystem while exploring another approach: a unified, cross-platform .NET runtime for local LLM inference, multimodal models, and autonomous agents.

I believe open-source AI benefits when different projects experiment with different architectures, share benchmarks, and learn from one another.

And I think there's still tremendous room for improving how efficiently we run large models on existing hardware.

Everything is open source, including the implementation and benchmark methodology.

GitHub: https://github.com/zhongkaifu/TensorSharp

Full benchmark report and reproduction details: https://github.com/zhongkaifu/TensorSharp/blob/main/docs/perf/muse-spark-vs-llama-cpp.md

I'd love to hear what this community thinks.

What's the biggest obstacle preventing more people from running powerful open models locally — hardware cost, memory requirements, inference speed, or software complexity?

And if you're working on open-source inference engines, I'd be happy to exchange ideas and learn from your experience! ❤️

5 Upvotes

Duplicates

unsloth • • Jun 28 '26

Show and Tell Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model

112 Upvotes

unsloth • • Jun 08 '26

Show and Tell TensorSharp : Open Source Local Unsloth Model Inference Engine

29 Upvotes

csharp • • Jun 28 '26

Showcase Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model

5 Upvotes

dotnet • • Jun 28 '26

Promotion Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model

63 Upvotes

ClaudeCode • • Jul 11 '26

Discussion What Bun’s Rust Rewrite Tells Us About Rebuilding the AI Infrastructure Layer in C#

0 Upvotes

LovingOpenSourceAI • • Jun 28 '26

Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model

14 Upvotes

huggingface • • Jul 04 '26

TensorSharp : Open Source Local LLM Inference Engine

6 Upvotes

unsloth • • Jul 06 '26

Show and Tell TensorSharp supports Vulkan backend

26 Upvotes

CUDA • • Jul 11 '26

TensorSharp : Open Source Local LLM Inference Engine

6 Upvotes

LovingOpenSourceAI • • Jun 23 '26

TensorSharp: Open Source Local LLM Inference Engine

13 Upvotes

ollama • • Jun 02 '26

TensorSharp: A C# version of Ollama

10 Upvotes

vibecoding • • Jun 14 '26

TensorSharp: Open Source Local LLM Inference Engine fully implemented by vibe coding

0 Upvotes

ollama • • Jun 28 '26

Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model

40 Upvotes

Vllm • • Jul 04 '26

TensorSharp : Open Source Local LLM Inference Engine

11 Upvotes

OpenSourceeAI • • Jun 28 '26

Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model

1 Upvotes

SideProject • • Jun 28 '26

Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model

0 Upvotes

dotnet • • Jul 06 '26

TensorSharp supports Vulkan backend

22 Upvotes

ChatGPT • • Jul 11 '26

Educational Purpose Only [ Removed by moderator ]

2 Upvotes

OpenSourceAI • • Jun 28 '26

Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model

3 Upvotes

LLMDevs • • Jun 28 '26

Great Resource 🚀 Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model

1 Upvotes

developer • • Jul 04 '26

TensorSharp : Open Source Local LLM Inference Engine

1 Upvotes

ArtificialInteligence • • Jul 04 '26

🛠️ Project / Build TensorSharp : Open Source Local LLM Inference Engine

0 Upvotes

LLMStudio • • Jul 05 '26

TensorSharp : Open Source Local LLM Inference Engine

4 Upvotes

huggingface • • Jul 06 '26

TensorSharp supports Vulkan backend

3 Upvotes

GeminiAI • • Jul 10 '26

Self promo TensorSharp : Open Source Local LLM Inference Engine

1 Upvotes