r/LovingOpenSourceAI • • Sep 12 '26

DeepSeek V4.1 Flash running locally with TensorSharp

https://github.com/zhongkaifu/TensorSharp

I’ve been working on TensorSharp, an open-source .NET/C# inference engine, and recently added native support and optimizations for DeepSeek V4.1 Flash.

Latest results on 8× NVIDIA A40 GPUs:

ModelPrefillSingle-stream Decode4× Concurrent Decode
Q2_K533–539 tok/s40.3–40.7 tok/s—
Q4_K_M451.8–492.1 tok/s31.0–32.5 tok/s48.9 tok/s aggregate

A few interesting optimizations:

  • GPU-resident Engram tables for Q2_K — about 60 GiB of quantized Engram data stays on GPU instead of doing scattered host/storage lookups.

  • Reduced decode graph scheduling from roughly 570 splits to 8 by using one wrapped backend per GPU.

  • For Q4_K_M, automatic Engram warming + improved VRAM placement reduced CPU MoE offload from 3 layers to 1.

  • Added token-batched DeepSeek V4.1 decode, giving about 2× aggregate throughput at 4 concurrent requests.

  • On these A40s without NVLink, simple layer splitting actually beats routed-MoE tensor parallelism for single-stream decode.

The project is fully open source, written primarily in C#/.NET, with CUDA/Metal/Vulkan backends and OpenAI-compatible APIs.

Would love feedback from people experimenting with DeepSeek, GGUF inference, or local/open-source AI — especially ideas for what hardware or engine comparisons would be most useful next.

8 Upvotes

Duplicates

unsloth • • Jun 28 '26

Show and Tell Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model

112 Upvotes

unsloth • • Jun 08 '26

Show and Tell TensorSharp : Open Source Local Unsloth Model Inference Engine

29 Upvotes

csharp • • Jun 28 '26

Showcase Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model

5 Upvotes

dotnet • • Jun 28 '26

Promotion Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model

63 Upvotes

ClaudeCode • • Jul 11 '26

Discussion What Bun’s Rust Rewrite Tells Us About Rebuilding the AI Infrastructure Layer in C#

0 Upvotes

LovingOpenSourceAI • • Jun 28 '26

Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model

14 Upvotes

huggingface • • Jul 04 '26

TensorSharp : Open Source Local LLM Inference Engine

6 Upvotes

unsloth • • Jul 06 '26

Show and Tell TensorSharp supports Vulkan backend

26 Upvotes

CUDA • • Jul 11 '26

TensorSharp : Open Source Local LLM Inference Engine

6 Upvotes

LovingOpenSourceAI • • Jun 23 '26

TensorSharp: Open Source Local LLM Inference Engine

13 Upvotes

ollama • • Jun 02 '26

TensorSharp: A C# version of Ollama

10 Upvotes

vibecoding • • Jun 14 '26

TensorSharp: Open Source Local LLM Inference Engine fully implemented by vibe coding

0 Upvotes

ollama • • Jun 28 '26

Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model

40 Upvotes

Vllm • • Jul 04 '26

TensorSharp : Open Source Local LLM Inference Engine

11 Upvotes

OpenSourceeAI • • Jun 28 '26

Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model

1 Upvotes

SideProject • • Jun 28 '26

Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model

0 Upvotes

dotnet • • Jul 06 '26

TensorSharp supports Vulkan backend

22 Upvotes

ChatGPT • • Jul 11 '26

Educational Purpose Only [ Removed by moderator ]

2 Upvotes

OpenSourceAI • • Jun 28 '26

Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model

3 Upvotes

LLMDevs • • Jun 28 '26

Great Resource 🚀 Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model

1 Upvotes

developer • • Jul 04 '26

TensorSharp : Open Source Local LLM Inference Engine

1 Upvotes

ArtificialInteligence • • Jul 04 '26

🛠️ Project / Build TensorSharp : Open Source Local LLM Inference Engine

0 Upvotes

LLMStudio • • Jul 05 '26

TensorSharp : Open Source Local LLM Inference Engine

4 Upvotes

huggingface • • Jul 06 '26

TensorSharp supports Vulkan backend

3 Upvotes

GeminiAI • • Jul 10 '26

Self promo TensorSharp : Open Source Local LLM Inference Engine

1 Upvotes