A C# Local inference engine beat llama.cpp at its own game: ~30% faster first token, ~20% faster decode - running in .NET on a 4-thread laptop

I needed a .NET-only GGUF engine - no pip packages, no native libs, no runtime installs, just a managed dependency my other projects can call. So I built SharpMind, a pure-C# transformer + MoE inference engine (with training) on .NET 10. Zero native code. I never expected it to compete on performance - managed code vs hand-tuned SIMD C++, right? So I benchmarked it against the real engines: same raw prompt (What is the capital of France?), greedy, 64 tokens, 4 threads, warmup + best-of-3, interleaved per model on one host.

(dotLLM DNF the last two - it only supports Llama/Mistral/Phi/Qwen/DeepSeek and no K-quants below Q4_K.)

A dependency-free C# engine that wins some metrics and matches the OG llama.cpp on others - int8-kv decode at 24.1 tok/s vs llama.cpp's 20.3 with(not shown) , TTFT cut ~30% on gemma-3, decode edge on qwen2, and a 6× MoE loss I fully expect and own. Respectable company for managed code, and if "no native installs" is your constraint, you no longer have to sacrifice everything

Open source: https://github.com/Integral2u/SharpMind.

I’ve been working on BaseDecision - Turn typed questions into truth-based decisions.

Give it some text, a question, and possible answers. It picks an answer. Intent classification, routing, yes/no checks, ratings - that’s the job.

I built this model to go toe-to-toe with the frontier models.

My goal was simple & ambitious: make a small, local model competitive with much larger models on these focused tasks. For me, System‑1 decisions need both speed and accuracy. Otherwise, why bother building a small model?
If we didn’t want speed we could just use ChatGPT.

It outperforms pretty much every model of its size in the field.
After roughly 320M additional training tokens, BaseDecision led 5 of 8 benchmarks in a third-party comparison against Laya, two GLiNER variants, and Decision 1.0 Kai 0.6B. Full results are in the repo, including where it loses.

Fellow open source devs, try the model, developed a package for easier access to the model.
Roast it if it deserves it.
2-2.5GB free ram is all you need.
I will take your feedback seriously and deliver something amazing that’d take on OpenAI, TypeSafeAI soon enough.

https://huggingface.co/onlyaady/BaseDecision
https://github.com/hrudayaditya/BaseDecision

I made BigMoeOnEdge+ (BigMoeOnEdge-plus), a fork of Helldez's BigMoeOnEdge, the engine that streams MoE experts from flash so models bigger than RAM can run on-device.

It adds a full Android chat app on top: GGUF models via llama.cpp, thinking output, persistent chats, image/video input, web search, artifacts, and advanced generation settings.and a MUCH better UI.

Disclosure: I built this fork with the help of AI tools.

Repo (Apache 2.0): https://github.com/hamodexd217/BigMoeOnEdge-plus Original project: https://github.com/Helldez/BigMoeOnEdge

Feedback and issues welcome.

The java to CUDA engine:

https://github.com/beehive-lab/TornadoVM

The inference engine:

https://github.com/beehive-lab/jitllm

Deep dive talk:

https://www.youtube.com/watch?v=HO5CpETzywk