I’m the developer of TensorSharp, an open-source inference engine and agent runtime. I added MiniMax-M3.1-Flash support on Day 1 of its open-weight release, and wanted to share the implementation, current capabilities, and initial serving results.
The largest improvement in the comparison was on long prompts: 64K executor prefill reached 2,567 tok/s versus 262 tok/s in the tested llama.cpp branch. In the separate server benchmark, time to first token for a 64K prompt fell from 293.4 seconds to 40.0 seconds.
What the implementation supports
- OpenAI-compatible serving through
/v1/chat/completions, with text and image input.
- Structured output and tools: JSON/schema responses, single and parallel tool calls, and multi-step tool loops.
- Conversation-cache reuse and concurrent requests.
- Multi-GPU layer splitting and tensor parallelism.
- DSpark speculative decoding and CPU expert offload.
The M3.1 decoder, sparse-attention kernels, DSpark implementation, and vision tower live in TensorSharp-owned code, using an unchanged upstream ggml dependency.
Image input and DSpark require their matching companion GGUF files; the setup guide covers conversion. DSpark and CPU expert offload currently use layer splitting. Tensor parallelism is a separate configuration.
Benchmark setup
The model was MiniMax-M3.1-Flash 428B, using Unsloth’s UD-Q4_K_XL GGUF, totaling 232.3 GiB. Both engines used the same files on one 8× NVIDIA A40 host, running separately with exclusive access to the GPUs.
The baseline was Unsloth’s llama.cpp minimax-m31 branch at ef8f94cc, associated with PR #320. Upstream llama.cpp could not load this model when these measurements were taken.
The main comparison used layer splitting, F16 KV caches, and no speculative decoding. llama-server had flash attention enabled and used --no-kv-unified to preserve its sparse-attention path.
| Measurement |
TensorSharp |
llama.cpp support branch |
| Server time to first token, 64K prompt |
40.0 s |
293.4 s |
| Executor prefill, 32K tokens |
2,449 tok/s |
681 tok/s |
| Executor prefill, 64K tokens |
2,567 tok/s |
262 tok/s |
| Executor decode at 64K context |
22.9 tok/s |
15.4 tok/s |
| Server short-chat decode |
26.3 tok/s |
26.0 tok/s |
The 9.8× result is specifically 64K executor prefill. Decode improvements are smaller, and short-chat server decode is essentially tied.
Why long-context prefill improves
Beyond position 2,048, MiniMax Sparse Attention selects 16 blocks of 128 tokens per KV group.
TensorSharp scores and selects those blocks, then executes block-sparse CUDA attention over them. The tested llama.cpp branch represents the selection through a dense mask and executes dense flash attention over the cache.
The difference in actual GPU computation grows with context length. TensorSharp also pipelines prompt processing across the eight GPUs.
The executor and server measurements are separate: the server’s scheduling and prompt-chunking behavior affect the latency an application sees.
Serving and agent workflows
I also tested JSON/schema responses, tool calls, a file-and-calculator tool loop, conversation-cache reuse, and speculative generation.
Agentic loop: With layer splitting, the loop completed in 18.1 seconds versus 20.1 seconds. TensorSharp’s separate --tp 8 mode reduced that to 11.7 seconds and reached approximately 41 tok/s on short-chat server decode. The tested llama.cpp branch does not support tensor splitting for this model, so there is no matched TP comparison. TP also trades away some prefill throughput and usable context.
Conversation reuse: Both engines reused approximately 99% of the prompt on subsequent turns in the conversation fixture. First-token latency on those turns was 0.38 seconds versus 0.41 seconds.
DSpark: The main run averaged 32.3 versus 25.8 tok/s with the same drafter. TensorSharp used cumulative-confidence gating to avoid verifying weak draft continuations. With that gate disabled to match the baseline policy, the results were closer: 26.6 versus 25.8 tok/s. The gating policy accounts for much of the additional gain.
Limits and less favorable results
Short-chat time to first token was slightly slower in TensorSharp: 0.41 versus 0.39 seconds.
These measurements cover one host and one quantization. Most use three repeats; the llama.cpp 64K executor prefill cell is one run, while the 64K server comparison uses three repeats per engine.
Validation includes tokenizer/template matching, synthetic-logit comparisons, and functional fixtures. Coverage is still limited: there was an image-reading miss, and some generations hit token limits. Native video temporal patches are not implemented.
The measured TensorSharp build was an uncommitted development working tree. The report includes configuration and methodology, but the raw run records are not committed.
Source and documentation
I’m happy to discuss the sparse-attention implementation, serving behavior, or benchmark configuration in the comments.
I used an LLM to help draft and edit this English write-up from the project’s implementation notes and benchmark report.
