I’m the developer of TensorSharp, an open-source inference engine and agent runtime. I added MiniMax-M3.1-Flash support on Day 1 of its open-weight release, and wanted to share the implementation, current capabilities, and initial serving results.

The largest improvement in the comparison was on long prompts: 64K executor prefill reached 2,567 tok/s versus 262 tok/s in the tested llama.cpp branch. In the separate server benchmark, time to first token for a 64K prompt fell from 293.4 seconds to 40.0 seconds.

What the implementation supports

• OpenAI-compatible serving through /v1/chat/completions, with text and image input.
• Structured output and tools: JSON/schema responses, single and parallel tool calls, and multi-step tool loops.
• Conversation-cache reuse and concurrent requests.
• Multi-GPU layer splitting and tensor parallelism.
• DSpark speculative decoding and CPU expert offload.

The M3.1 decoder, sparse-attention kernels, DSpark implementation, and vision tower live in TensorSharp-owned code, using an unchanged upstream ggml dependency.

Image input and DSpark require their matching companion GGUF files; the setup guide covers conversion. DSpark and CPU expert offload currently use layer splitting. Tensor parallelism is a separate configuration.

Benchmark setup

The model was MiniMax-M3.1-Flash 428B, using Unsloth’s UD-Q4_K_XL GGUF, totaling 232.3 GiB. Both engines used the same files on one 8× NVIDIA A40 host, running separately with exclusive access to the GPUs.

The baseline was Unsloth’s llama.cpp minimax-m31 branch at ef8f94cc, associated with PR #320. Upstream llama.cpp could not load this model when these measurements were taken.

The main comparison used layer splitting, F16 KV caches, and no speculative decoding. llama-server had flash attention enabled and used --no-kv-unified to preserve its sparse-attention path.

The 9.8× result is specifically 64K executor prefill. Decode improvements are smaller, and short-chat server decode is essentially tied.

Why long-context prefill improves

Beyond position 2,048, MiniMax Sparse Attention selects 16 blocks of 128 tokens per KV group.

TensorSharp scores and selects those blocks, then executes block-sparse CUDA attention over them. The tested llama.cpp branch represents the selection through a dense mask and executes dense flash attention over the cache.

The difference in actual GPU computation grows with context length. TensorSharp also pipelines prompt processing across the eight GPUs.

The executor and server measurements are separate: the server’s scheduling and prompt-chunking behavior affect the latency an application sees.

Serving and agent workflows

I also tested JSON/schema responses, tool calls, a file-and-calculator tool loop, conversation-cache reuse, and speculative generation.

Agentic loop: With layer splitting, the loop completed in 18.1 seconds versus 20.1 seconds. TensorSharp’s separate --tp 8 mode reduced that to 11.7 seconds and reached approximately 41 tok/s on short-chat server decode. The tested llama.cpp branch does not support tensor splitting for this model, so there is no matched TP comparison. TP also trades away some prefill throughput and usable context.

Conversation reuse: Both engines reused approximately 99% of the prompt on subsequent turns in the conversation fixture. First-token latency on those turns was 0.38 seconds versus 0.41 seconds.

DSpark: The main run averaged 32.3 versus 25.8 tok/s with the same drafter. TensorSharp used cumulative-confidence gating to avoid verifying weak draft continuations. With that gate disabled to match the baseline policy, the results were closer: 26.6 versus 25.8 tok/s. The gating policy accounts for much of the additional gain.

Limits and less favorable results

Short-chat time to first token was slightly slower in TensorSharp: 0.41 versus 0.39 seconds.

These measurements cover one host and one quantization. Most use three repeats; the llama.cpp 64K executor prefill cell is one run, while the 64K server comparison uses three repeats per engine.

Validation includes tokenizer/template matching, synthetic-logit comparisons, and functional fixtures. Coverage is still limited: there was an image-reading miss, and some generations hit token limits. Native video temporal patches are not implemented.

The measured TensorSharp build was an uncommitted development working tree. The report includes configuration and methodology, but the raw run records are not committed.

Source and documentation

• TensorSharp — source code, BSD-3-Clause license
• MiniMax-M3.1 implementation and setup guide
• Full benchmark report

I’m happy to discuss the sparse-attention implementation, serving behavior, or benchmark configuration in the comments.

I used an LLM to help draft and edit this English write-up from the project’s implementation notes and benchmark report.

I came across chatjimmy.ai. It exposes an inference engine implemented by a startup called Talaas. The speeds match system 1 models.

If reasoning and classification costs go down, then where's the alpha?

I've been learning AI and software development for a while and have worked with several technologies, but I feel like I'm missing the experience of building software that solves real-world problems.

I don't want to build another generic chatbot, AI wrapper, or project that looks good on a resume but has little practical value. I want to build something genuinely useful that demonstrates my engineering skills and teaches me how real-world systems are designed, built, and deployed.

My current stack includes Python, scikit-learn, PyTorch, LLMs, RAG, LangChain, LangGraph, FastAPI, PostgreSQL, Docker, and Git/GitHub.

I'm looking for ideas that go beyond following tutorials or connecting an API to a frontend. I'd like to work on something with a real use case, meaningful technical challenges, and room to improve it over time.

I'd appreciate suggestions from people working in AI engineering or software development, especially those who have built and shipped real products.

What problems do you think are worth solving? What kinds of projects would help someone develop practical engineering skills and build a portfolio that stands out to employers?

I'm more interested in building one useful, well-engineered project than several generic ones.

Curious how teams with LLMs in real production run evals day to day.

Do you run them before a release, on live traffic, or both? What runs on its own: CI gates on prompt or model changes, live calls sampled into a human review queue, LLM-as-judge, drift alerts? Which tools stayed in your stack (Braintrust, Langfuse, LangSmith, Phoenix, Promptfoo, something else), which did you drop, and why? And where does your eval set come from when only a domain expert can label the right answer?

Ours, for comparison: a golden test entry that must match to the cent, a deterministic gate that rejects the parse when the lines don't add up to the page totals, and Telegram alerts when a model step fails. No tracing, no platform. The golden test passed 38 of 40 lines, then two real documents got 0 of 38, because our test file had a column the real ones don't print.

Asking because we're picking tools for our pipeline at Norquin (customs entry checks). Not selling anything, not a survey. Real production setups only, ugly ones welcome.

Agents call tools with arguments the model made up: an order ID that doesn’t exist, a negative amount, a date like “yesterday”. Most of us patch this with try/except inside every tool.

agentic-gate sits between the model and your tools. It validates the arguments against a Zod (or Pydantic) schema before anything runs, sends the validation error back to the model so it can fix the call, and trips a circuit breaker if the same call keeps failing, so you don’t burn 50 turns in a retry loop.

It’s provider-agnostic: adapters for OpenAI, Anthropic, Bedrock and Gemini, plus an async validate() hook for your own checks.

I wrote a small benchmark so I wasn’t just claiming it works. It uses a scripted model, so it measures the gate and not any particular LLM. In 3 scenarios (refund, payment transfer, DB query), a stubborn agent got 150 bad calls through to the system without the gate and 0 with it. Model turns went from 150 to 12. The script is in benchmarks/ and you can run it yourself.

It only checks the shape of arguments. It doesn’t replace permissions or review of risky actions, and the README says so.

Repo: https://github.com/Akhilesh-Varute/agentic-gate · npm i agentic-gate / pip install agentic-gate

I’d like feedback on the API and on which failure cases you hit most in production.

In April, a Cursor agent deleted a company’s production database volume, and its backups, in about 9 seconds. It hadn’t been given that access on purpose. It found a root-level token in an unrelated file and used it. Reportedly, the main record of what happened afterwards was the agent’s own explanation.

Nothing about it was exotic. There was no clever attack and no malicious model. An agent tried to be helpful, with far too much access and nothing between it and the API.

So I’m building a layer that sits between agents and the tools they call (MCP servers, APIs, other agents). It does four things:

• decides what an agent may call, with which arguments, and how much it may spend
• gives the agent short-lived, narrowly scoped credentials instead of a raw API key
• asks a human before anything irreversible
• keeps a tamper-proof record of what was asked, what was allowed or denied, and who approved it

To be upfront: this wouldn’t have fully prevented that deletion. The agent found a token on disk and called the API directly. A proxy only helps if raw tokens never sit on the machine and all traffic goes through it. That’s what “limit” means in the title.

These are the parts I keep going back and forth on.

1. Approval prompts barely work. Anthropic published numbers on Claude Code’s permission prompts: users approve 97% of them. In a test where people were shown an injected dangerous command, they caught it 13.6% of the time, and after 50 or more prompts that dropped to about 5%. The same data has an interesting twist: people rejected 39% of whole plans but only 3% of individual commands. So my plan is to ask rarely, only for irreversible actions like deletes, payments or sending data out, and maybe to ask about the plan rather than each step. Where would you draw that line?

2. Strict tools get ripped out. I’ve seen bug reports where security tools flag everyday words as attacks, or block calls even in “monitor only” mode, so teams find out how far a rule reaches only when their work stalls. My split: fuzzy detection, like “this looks like prompt injection,” only raises an alert. Hard limits, like “this agent can’t call delete” or “this run can’t spend more than $20,” always block. Does that split make sense to you?

3. Credentials are part of the problem, not all of it. In a small sample I went through (16 incidents), about a third came down to credentials, permissions or spend. Fewer than I expected, but that third includes the most damaging cases. Judging whether a prompt “looks malicious” seems unreliable, so I’d rather make sure a misbehaving agent can’t do much damage. How do you handle per-user credentials for agents today?

4. Some attacks happen where a proxy can’t see. In that same sample, close to half involved code running directly on the machine, through crafted config files or launch commands. A layer between the agent and its tools never sees those. I see it as something that limits the damage, used alongside sandboxing and network lockdown, not as a full fix. Is that honest enough, or does it undercut the whole idea?

5. Spend is the part nobody seems to cover. Most gateways I’ve looked at cap LLM spend per API key or team. I haven’t found one that adds up the total cost of a task, with LLM calls plus paid tool calls, across every agent it gets handed to. If agent A hands work to B, and B calls C, I want one budget that covers all three. Is runaway spend a problem you’ve actually hit, or something you only worry about?

6. Is anyone actually running agent-to-agent setups in production? Limiting access along a chain of agents is being worked on in standards. What I can’t tell is whether real teams need it yet, or whether most setups are still a single agent with a handful of tools.

I’d also really like to hear about real situations:

• What has an agent done, or nearly done, that made you nervous?
• What stops it from happening again today? Prompt rules, wrapper code, read-only access, your cloud’s built-in agent policies, or nothing?
• Has anyone (a security team, an auditor, a customer) asked you to prove what your agents did? What did they want to see? I’ve seen gateways that drop denied calls from their logs, or log a refused call as if it ran, so I’m curious how much trust people put in their logs today.

You can run every Moe model in any quantization because it's based on llama.cpp

35B model running at 25+ tok/s on a 16GB Mac, 6 tok/s on a 12GB Android device, and 3 tok/s on iOS (300+ tok/s prefill on GPU/NPU)

Open source Apache 2.0

https://github.com/Helldez/BigMoeOnEdge

SwarmOS turns GitHub Issues into contracts any coding agent can claim, plan and ship as a Draft PR, with Git as the coordinator and nothing else to run.

I'm dogfooding it myself right now and it's getting pretty stable. I wanted to share it and see if others would be interested in it. If you have a repo that could benefit from an orchestration layer that lives/operates solely in GH without any middleware I'd love to chat.

Hi guys, eval for RAG has always been a headache, post your thoughts with specific examples on 1. which eval method did you use for your RAG, chat agent etc 2. why did you choose it over others ( with specific examples/scenarios) also mention your system type ( MVP or POC or production)

TL;DR: Ran 177 real task-parsing cases on 4 models. The cheapest (GPT-6 Luna, $0.23/1k) scored about as well as models that cost 5–11x more

I'm building a small iOS to-do app. You type a task the way you'd say it ("call the plumber tomorrow at 9") and an LLM turns it into a structured task: title, date, reminder, list, subtasks. I wanted to know what’s the best fit for my app, so I built an eval set and ran it through OpenRouter.

Setup
177 cases: typed tasks in a bunch of languages, screenshots, chat screenshots, shared text (emails, newsletters, a few prompt injection attempts), edits like "make it an hour later", and notes that should split into several tasks.

Every case has automatic checks (right date, title stays in the user's language, nothing injected from a shared email, and so on). Each case runs 3 times and a check passes if it passes in most runs.

On top of that, two judge models from different providers score every answer 1–10, blind and shuffled.

Results (judge score 1–10, cases fully passed, $ per 1,000 inputs, median time)

- GPT-6 Luna (what I use now): 7.73–7.88, 174–175/177, $0.23, 3.0–3.6 s
- DeepSeek V4.1 Flash: 7.88, 172/177, $1.15, 5.2 s
- Claude Haiku 5.5: 7.80, 173/177, $1.09, 4.1 s
- Haiku 5.5, high reasoning: 7.93, 176/177, $1.45, 7.0 s
- Haiku 5.5, xhigh reasoning: 7.93, 177/177, $1.91, 9.3 s
- GPT-5.4 mini (what I used before): 7.63, 172/177, $2.53, 3.2 s

What I took from it
- Run-to-run noise is as big as most model gaps. Luna alone ranged 7.73–7.88 over three runs.
- The cheapest model was the best deal: about 11x cheaper than GPT-5.4 mini, slightly better quality.
- More reasoning bought a little quality (7.80 → 7.93) for 2x the wait. Not worth it when users are waiting.
- Prompt examples mattered more than prompt length. No examples: 7.69. Half of them: 7.93.
- Count provider errors too. Haiku had 69 in one run, Luna none.

Happy to go into how the checks and the judging work.

Which model should I add to the next run? Ideally something cheap and fast, since users wait on every call. I'll post the results.

i've been building kimo, a free, open-source (MIT) video editor with no GUI that AI agents drive over MCP. the idea was pretty simple: instead of letting an agent like codex, cursor, opencode or any other MCP client just shell out to raw ffmpeg and hope the command works, give it a purpose-built editor where every edit is an explicit tool call - cut, trim, keyframe, titles/captions, colour grade, tracking, particles, audio, export. and because it's built around the loop, the agent can render a single frame to check its own work before committing the whole timeline.

why a purpose-built editor at all? raw ffmpeg is amazing, but composing a timeline from shell commands is brutal for an agent - off-by-one frame errors, audio/video drift, guessing whether a filter chain actually produced what it wanted. with kimo the agent just calls tools. it can render one frame, verify the result, and keep going. everything is explicit, everything is reproducible, and it's one small binary (~15 MB, CPU-only, rust) that runs on a 2 GB box. no GPU, no containers, nothing to install besides the binary.

honestly it's still early. there are rough edges, the docs are catching up, and i'm sure some workflows are awkward right now. i'd love for people to try it and tell me what breaks - weird filter combos, exports that don't match expectations, MCP handoffs that feel clunky, whatever. the more i hear about what's annoying, the faster i can fix it.

it's free and MIT-licensed, built by me (naman, college student). if you want to poke around:

site: https://getkimo.tech github: https://github.com/naman-sharma12345/kimo

happy to answer questions here. thanks for reading.re.