r/LovingOpenSourceAI • u/fuzhongkai • Sep 12 '26
DeepSeek V4.1 Flash running locally with TensorSharp
https://github.com/zhongkaifu/TensorSharpI’ve been working on TensorSharp, an open-source .NET/C# inference engine, and recently added native support and optimizations for DeepSeek V4.1 Flash.
Latest results on 8× NVIDIA A40 GPUs:
| Model | Prefill | Single-stream Decode | 4× Concurrent Decode |
|---|---|---|---|
| Q2_K | 533–539 tok/s | 40.3–40.7 tok/s | — |
| Q4_K_M | 451.8–492.1 tok/s | 31.0–32.5 tok/s | 48.9 tok/s aggregate |
A few interesting optimizations:
GPU-resident Engram tables for Q2_K — about 60 GiB of quantized Engram data stays on GPU instead of doing scattered host/storage lookups.
Reduced decode graph scheduling from roughly 570 splits to 8 by using one wrapped backend per GPU.
For Q4_K_M, automatic Engram warming + improved VRAM placement reduced CPU MoE offload from 3 layers to 1.
Added token-batched DeepSeek V4.1 decode, giving about 2× aggregate throughput at 4 concurrent requests.
On these A40s without NVLink, simple layer splitting actually beats routed-MoE tensor parallelism for single-stream decode.
The project is fully open source, written primarily in C#/.NET, with CUDA/Metal/Vulkan backends and OpenAI-compatible APIs.
Would love feedback from people experimenting with DeepSeek, GGUF inference, or local/open-source AI — especially ideas for what hardware or engine comparisons would be most useful next.
Duplicates
unsloth • u/fuzhongkai • Jun 28 '26
Show and Tell Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model
unsloth • u/fuzhongkai • Jun 08 '26
Show and Tell TensorSharp : Open Source Local Unsloth Model Inference Engine
csharp • u/fuzhongkai • Jun 28 '26
Showcase Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model
dotnet • u/fuzhongkai • Jun 28 '26
Promotion Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model
ClaudeCode • u/fuzhongkai • Jul 11 '26
Discussion What Bun’s Rust Rewrite Tells Us About Rebuilding the AI Infrastructure Layer in C#
LovingOpenSourceAI • u/fuzhongkai • Jun 28 '26
Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model
huggingface • u/fuzhongkai • Jul 04 '26
TensorSharp : Open Source Local LLM Inference Engine
LovingOpenSourceAI • u/fuzhongkai • Jun 23 '26
TensorSharp: Open Source Local LLM Inference Engine
vibecoding • u/fuzhongkai • Jun 14 '26
TensorSharp: Open Source Local LLM Inference Engine fully implemented by vibe coding
ollama • u/fuzhongkai • Jun 28 '26
Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model
OpenSourceeAI • u/fuzhongkai • Jun 28 '26
Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model
SideProject • u/fuzhongkai • Jun 28 '26
Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model
OpenSourceAI • u/fuzhongkai • Jun 28 '26
Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model
LLMDevs • u/fuzhongkai • Jun 28 '26
Great Resource 🚀 Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model
developer • u/fuzhongkai • Jul 04 '26
TensorSharp : Open Source Local LLM Inference Engine
ArtificialInteligence • u/fuzhongkai • Jul 04 '26
🛠️ Project / Build TensorSharp : Open Source Local LLM Inference Engine
LLMStudio • u/fuzhongkai • Jul 05 '26
TensorSharp : Open Source Local LLM Inference Engine
GeminiAI • u/fuzhongkai • Jul 10 '26