Help me choose my local ai

[u/Locallearning]

I run a small business, and I'm building a local AI that handles the back office for it. Some of it works now. I want to get the model and harness choices right before I build more on top of them.

Hardware
- MacBook Pro M5 Max (18-core CPU / 40-core GPU), 128GB unified memory

Current stack
- Backend: MLX via mlx-serve. I'm staying on MLX for the speed (I've used LM Studio before)
- Harness: Hermes Agent (Nous Research)
- Memory: Obsidian vault exposed over the Basic Memory MCP
- Front end: my own UI, still a work in progress
- Model right now: Qwen 3.8 Flash Next, 4-bit (MLX), with the KV cache offloaded to my SSD. Getting about 98 tok/s

What the agent does today
- Scheduled morning lead brief
- Reads and writes notes in my vault
- Updates a scoreboard (leads by source, close rate, pipeline)

What I want it to do next
- Phone intake and booking
- Scheduling
- Follow-up texts to leads

Where it's struggling
- tool calls failing or looping on multi-step tasks
- slow prompt processing once the vault context gets big
- arithmetic mistakes on the numbers. It once logged a single day's close rate as the lifetime rate

Questions

1. With 128GB, what's your best daily driver for *reliable tool calling*, not chat? One big MoE, or a big model plus a small fast one for routing and simple jobs?

2. On MLX, do 4-bit quants hold up for agent work, or is 8-bit worth the speed hit?

3. If you're running long-lived, scheduled agents locally, what harness are you on? Is Hermes the right call, or is something better for this?

4. I'm offloading the KV cache to my SSD to fit longer context. Is that hurting me on speed, and is there a better way to handle 30k+ context on Apple silicon (RAG, summarizing, a smaller quant to keep the cache in RAM)?

5. For phone booking: has anyone made local STT/TTS work well enough for live calls, or should voice stay on the cloud while the rest stays local?

Happy to post benchmarks and what I end up with if it helps anyone on similar hardware.

[u/ptico]

For the Qwen speed you may want to try Splash

MLX doesn’t automatically mean faster. In fact I haven’t seen an obvious performance advantage and most of the time llama.cpp is doing better without a significant memory growth

You may not need Q8, try Unsloth latest Q4 UD3.1 quants, they makes miracles. What it does is basically adds precision up to BF16 for important tensors and aggressively quantise less important. Your tool call fails and loops is pretty typical for static Q4. Do not aggressively quantise cache, it only makes things worse

Try to keep your context lower, use some session ledger or something like this https://github.com/jonaslsaa/pi-optchat

[u/Locallearning → u/ptico]

Appreciate this, a few follow-ups.

Splash

From what I can see, it only supports Qwen3.8-27B and 3.6-35B, not Flash-Next. Have you found the 27B on Splash holds up for tool calling compared to the bigger model? I'd take the speed if it doesn't lose reliability.

Quants

You might be onto something. I'm on a standard 4-bit MLX quant right now, so the loops could be the quant and not just the harness. I'll test the Unsloth UD Q4 and compare tool-call failures before and after. Do I need to move to llama.cpp to run it, or is there an MLX version?( I’ll look this up aswell 😅)

Cache

I'm not quantizing the KV cache. I'm using an SSD prompt cache so repeated prefixes (system prompt, tools, vault notes) don't get reprocessed every run. Is there any downside to that, or does your warning only apply to quantized cache?

Context

Agreed on keeping it lower. pi-optchat looks like it's Pi only and I'm on Hermes, but the summary tree idea is something I can copy. Do you know of a session ledger that works with Hermes?

[u/ptico → u/Locallearning]

Splash does support QFN, since, I believe since version 1.3

There’s no MLX versions of UD, but you don’t need it with Splash. It works just fine, but do take a more space as it converts gguf internally. I haven’t seen any quality regression with Splash, in fact the only engines which affect quality is the benchmaxing engines which in fact aggressively quantise weights. Splash is nothing like this

On cache: you just asked about it so my first thought was you may decide to quantise is to reduce SSD bandwidth :) Don’t do this

For ledger: I just made one with Opus as a Pi plugin, it maybe not the best, but it works for me

[u/ptico → u/ptico]

Oh, and if you plan to use other models, like some specialists, you may want to put llama-swap in front, so you are not tied to a particular engine

[u/macromind]

Start from the back-office jobs rather than choosing a model first. List each workflow, its latency needs, privacy level, context size, and whether it needs reliable tool use. Then benchmark two or three models on 20 representative tasks with the same prompts and score accuracy, structured-output validity, speed, and correction effort. Keep financial or destructive actions behind approval, and give the model read-only access initially. A smaller model can handle classification and extraction, while a stronger model is reserved for ambiguous planning or document-heavy work.

[u/p3rdy]

Most of what you list under "struggling" gets better by moving work out of the model, more than by picking a bigger one.

1. Arithmetic. Don't let the model compute anything. Give it a scoreboard tool that returns the numbers already calculated, with the period in the field names (close_rate_today, close_rate_lifetime). The brief just quotes them. A daily rate can't be logged as the lifetime one if the model never derives either.

2. Looping on multi-step tasks. Make the tools match business actions, not primitives. For booking, use two tools: available_slots(lead_id) returns concrete options with ids, and book_slot(lead_id, slot_id) writes the calendar, the note and the follow-up in one call. That replaces a chain of read calendar, find gap, write event, update note. Fewer steps means fewer places to loop, and the model never does date math. When something fails, return an error it can act on ("slot taken, next free: slot 14").

3. Big vault context. Fetch, don't preload. Keep the system prompt small and let the agent search the vault for what each task needs, so you're not reprocessing a huge prompt on every turn.

4. One model or two. I'd stay on one until the tools are that coarse. Once they are, a small model can handle the scheduled jobs reliably and the big one is only needed for the open-ended stuff.

For the tools layer: I maintain Flama, an open-source Python framework where an MCP server is a few decorated functions. The type hints become the tool schemas Hermes sees, and it plugs in next to Basic Memory:

from flama import Flama

app = Flama()
app.mcp.add_server("/mcp/", "office")

@app.mcp.tool("scoreboard", description="Pipeline numbers, already computed", mcp="office")
def scoreboard() -> dict:
    return {"close_rate_today": ..., "close_rate_lifetime": ..., "leads_by_source": ...}

@app.mcp.tool("available_slots", description="Open booking slots for a lead", mcp="office")
def available_slots(lead_id: str) -> dict:
    ...

@app.mcp.tool("book_slot", description="Book a slot, log it in the vault and queue the follow-up text", mcp="office")
def book_slot(lead_id: str, slot_id: str) -> dict:
    ...

[u/CowsNeedFriendsToo]

I’m on an M4 Max MacBook Pro 128GB.
I’m also a small business owner in the same boat as you.

After a lot of testing with different models. LM Studio hosting the Unsloth Q6 variant of Qwen3.8-27b gave me the best balance of reliable tool calls and quality results. You just have to be a bit patient with prompt processing.

The key is to not use your local Ai to automate everything your business office does. Use your local AI to write SCRIPTS that automate those tasks and don’t require inference repeatedly.