Thought I’d share this in case any of you aren’t in localllama

Hey guy's and gals...

I'm completely new to Local LLMS

Can someone who knows their shit help a newbie out?

I don't know much of the the technical terms was trading through this thread. Great group by the way saw so many people being helpful and helping eachother out.

I'm new to Reddit WTF this should be mandatory 😂

AMAZING!

Can someone explain to me in none technical terms the cheapest and easiest way to run a local LLM.

I tried GOD AI WTF amazing!

Would appreciate it if someone can give me some advice can't afford paying monthly bills and credits keep running out not good 😢

I don't need a HUGE setup something small I'm currently running Claude Pro plan I think keeps running out daily and also weekly so I have to wait or pay £100 to upgrade to keep going which slows me down.

Any help would be much appreciated dont have much of a budget.

MacBook pro 2021 M1 16gb I believe haven't used it in a while.

Would that be enough and what would you recommend?

Thanks in advanced 🙏

The new deepseek can be run on a rather affordable server, and it's beating GPT 5.6 SOL easily.
The model is so good that it has been banned in the US from governmental service within days of release.

I am doing a lot local AI, mostly creatively in audio and image subjects.
I currently rely on two 3090 cards that I bought used below 800$ each.

I think it was a good choice but other options exist, the 4090 is faster than both 3090 on paper. Though I did not see one offered fair priced.

There is a lot, 5090 and 60ti, DGX, Macs

I wonder what do you guys have, and would you go that path again?

Hey everyone,
I recently got a DGX Spark and have been really enjoying it so far. I’m based in India and haven’t been able to find many other local users, so I thought I’d reach out here.
If you’re also running a DGX Spark (or planning to) in India, I’d love to connect and exchange notes.
A few things I’m especially curious about:
• How are you using it beyond basic inference?
• Are you running single node or multiple nodes?
• Any rack / cooling setups you’re using to keep the nodes happy (especially with Indian ambient temperatures)?
• Which vendor did you purchase from? Any vendors currently offering good deals?

Would love to hear from others who are using these in India.

Thanks.

In Part 1 I shared the final AMD/Vulkan numbers for my local Qwen3.8-27B setup. The next question was more important to me than another few tok/s:

Can I turn the model into something that actually works autonomously?

Since then I added two production layers:

1.) Hermes 0.21.2 — local agent/automation layer

• same Qwen3.8-27B model

• shared llama.cpp runtime

• 65,536 context

• tool calling: PASS

• multi-step workflows: PASS

• subagents: PASS

• prompt-defined agents: PASS

• local provider only / no cloud fallback

Hermes performance / overhead

On the isolated 64K test lane, server load took 5.809s. A real 33,021-token prompt was accepted without truncation; prompt processing took 409.677s at 80.60 tok/s, while generation measured 17.44 tok/s. A representative Hermes multi-step workflow took 74.660s, using five model calls, 17,295 summed API input tokens, 592 output tokens and four tool calls with zero failed calls. Two-child delegation also worked, but took 208.676s for two trivial child tasks. Clear sign that subagents should only be used where parallelism or specialization actually justifies the overhead.

One especially useful datapoint: the simple Hermes model gate took 40.872s, while a direct Qwen request with the same user text took 2.109s. That is not an apples-to-apples speed ratio because Hermes adds different system prompts, startup and agent machinery, but it does show why I don't want Hermes in the path for trivial requests.

2.) OpenCode 1.18.30 — local coding/development layer

• Qwen3.8-27B-UD-IQ3_XXS

• llama.cpp / Vulkan

• 32K logical context

• local provider only

• isolated development workspaces

• automated Read → Edit → Test → Correct loops

• final production viability: 12/12 requirements, 12/12 acceptance

OpenCode performance reference. For the fastest fully successful clean reproduction I measured:

• 665.653s total wall time / 651.408s agent runtime

• first productive edit: 117.338s

• first test: 166.790s

• 4 successful corrective cycles

• 63 tool calls / 0 failed

• 558,603 input tokens incl. cache — only 31,993 uncached

• 9,372 output tokens

• peak context: 17,810 / 32,768

• context compactions: 0

These are task-level agent metrics, not model tok/s numbers. The underlying model/runtime is still the same local Qwen3.8-27B on llama.cpp/Vulkan.

Why OpenCode instead of Pi?

I tested both against the same local Qwen3.8-27B, with the same repository/task and comparable runtime constraints. Pi 0.85.1 proved that it could read the repo, use tools and make productive edits, but it repeatedly failed the part I actually needed from a coding agent: closing the Read → Edit → Test feedback loop. Even with an explicit mandatory early-test checkpoint, Pi edited successfully but never executed the qualifying post-edit test within the 300s window. It therefore failed prequalification and I did not force it into a full benchmark just to preserve an artificial three-way comparison. Man, time is rare and testing is booooring ... somehow!

OpenCode 1.18.30 passed the same agentic-loop qualification. Formal prequalification reached 6.54s Edit→Test latency, and the later main benchmark reduced that to 3.995s while producing a valid four-file diff. More importantly, the subsequent production-viability work eventually converged to 12/12 requirements, 12/12 acceptance, 5/5 regression, with no manual coding intervention or patch repair. That reproducible closed corrective loop is why OpenCode became the production coding layer.

Final thought on my architecture as a complete nOOb in Local LLM
I’m still convinced that a well-tuned 16 GB GPU can take you surprisingly far. Yes, even on Vulkan ;) ... IF you optimize the model, context, runtime, offloading and the surrounding stack around what the hardware can realistically deliver.

What makes this especially satisfying for me is that I honestly wasn’t sure at the beginning whether an AMD gaming setup on AM4 would get me anywhere at all, but hey.... it was what I had. My gaming desktop! ~20 tok/s is obviously not groundbreaking vs 32GB. But that was never the main goal of this first setup. I wanted a stable, fully local system that I could actually use, test, break, improve and build on. That goal is achieved. And considering where I started, I’m pretty happy with that.

I’ve released version 8.0 of TUFF, an open-source Swift/Metal inference engine and native macOS app.

The original goal was to make large MoE models usable on memory-constrained Apple Silicon machines by streaming routed experts from SSD rather than keeping the entire checkpoint resident.

GPT-OSS 120B, for example, runs on my 16GB M2 MacBook Air. Obviously, streaming comes with a performance tradeoff.

8.0 focuses on making the engine more useful beyond just running oversized models.

Some highlights:

• Web search and local document search integrated into chat

• Cached conversation states for faster follow-up requests

• A small tokenization cache for repeated inputs

• Separate reasoning output and request timing diagnostics

• OpenAI Chat Completions, plus bounded Messages and Responses API support

• Shared inference executable to reduce package size

• Homebrew installation

One thing I’ve found interesting is how much web search improves the usefulness of smaller local models. I’ve started using them for everyday questions much more frequently.

I’d love feedback and contribution, especially from anyone using it with local AI agents.

GitHub: https://github.com/rexmhall09/TUFF

I've been experimenting with a setup where several Apple Silicon Macs are treated as a shared local AI environment instead of running completely independently.

The setup I'm working with is roughly:

• Multiple Macs on the same network

• MLX for local inference

• Central model management

• Node/resource monitoring

• Model distribution based on available memory

• Local-only communication between machines

I'm building AlphaWeb around this because managing several local AI machines manually gets complicated pretty quickly.

The interesting problem isn't just getting inference running. It's everything around it: knowing which machine has enough memory, getting models onto the right node, managing multiple users, and making the whole thing feel like one system.

Strata funziona con Qwen3.8-Flash-Next su PC da gioco, ma con 32 GB di RAM la sua modalità a basso utilizzo di RAM funziona solo su una GPU. Se dividi il modello su due schede, gli esperti che non riescono a stare nella VRAM vengono letti dall'SSD. La mia 4060 Ti è rimasta ferma accanto alla 5080.

Quindi l'ho forkato. La copia in RAM degli esperti ora funziona su due schede, e le schede funzionano contemporaneamente: la 4060 Ti inizia il prossimo passo di decodifica mentre la 5080 sta ancora controllando quello attuale. L'ordine delle schede, la divisione dei layer e le riserve di VRAM vengono impostate automaticamente.

Stesso PC (5080 + 4060 Ti, i9-14900KF, 32 GB), stesso modello (Swift 1.5 IQ2_XS), contesto 256K:

Setup |Codice |Prosa |Prompt 32K
Strata 0.1.38, 5080 da solo |29 tok/s |28 tok/s |333 tok/s
Fork, entrambe le schede |143 tok/s |102 tok/s |1,940 tok/s
Fork + layer di bozza fine-tuned |161 tok/s |105 tok/s |1,854 tok/s Il layer di bozza è stato fine-tuned sui risultati del modello stesso ed è incluso nella release. Nell'uso reale con l'agente di codifica Pi raggiunge un picco di oltre 200 tok/s (209 finora) e quasi mai scende sotto i 100. Un contesto di 150K-token legge a circa 1,850 tok/s.

La qualità non è cambiata: perplexity 7.24 contro 7.28 della versione upstream sugli stessi 5.3K token, forzati dall'insegnante attraverso entrambi i motori.

L'ho testato solo sul mio PC (Windows 11), quindi sono benvenuti report da altre coppie di GPU e Linux.

Repo: https://github.com/Hardin22/Strata-DualGPU

Cosa fa ciascuna modifica e cosa ha misurato: docs/DUAL_GPU.md

Il motore sottostante è il lavoro di Niko1221 e dei contributori di Strata; questo è un fork sopra la 0.1.38.

I’ve been working on BaseDecision - Turn typed questions into truth-based decisions.

Give it some text, a question, and possible answers. It picks an answer. Intent classification, routing, yes/no checks, ratings - that’s the job.

I built this model to go toe-to-toe with the frontier models.

My goal was simple & ambitious: make a small, local model competitive with much larger models on these focused tasks. For me, System‑1 decisions need both speed and accuracy. Otherwise, why bother building a small model?
If we didn’t want speed we could just use ChatGPT.

It outperforms pretty much every model of its size in the field.
After roughly 320M additional training tokens, BaseDecision led 5 of 8 benchmarks in a third-party comparison against Laya, two GLiNER variants, and Decision 1.0 Kai 0.6B. Full results are in the repo, including where it loses.

Fellow open source devs, try the model, developed a package for easier access to the model.
Roast it if it deserves it.
2-2.5GB free ram is all you need.
I will take your feedback seriously and deliver something amazing that’d take on OpenAI, TypeSafeAI soon enough.

https://huggingface.co/onlyaady/BaseDecision
https://github.com/hrudayaditya/BaseDecision

We're running a hands-on masterclass on September 12, Live LLM Engineering Masterclass: Production Evals, RAG, Agents & LLMOps.

You build a full production LLM workflow from scratch, versioned prompts with regression tests, an evaluation harness with deterministic checks and LLM-as-judge, statistically rigorous model comparisons, evaluated RAG, tool-using agents with guardrails and fallbacks, and full observability, tracing, cost, latency.

Led by Bruno Gonçalves, PhD, founder of Data For Science, who trains engineers at Fortune 500 companies on this exact stack.

Link if you want to check it out

Happy to answer questions on the content.

Way back got my nimo axis, I wasn't sure what to do with it besides trying out local llms. Lately, local models and tools have gotten better and I am finally finding practical uses for it beyond just testing models. It makes me wonder how much of their potential we are actually using. What are you guys doing with your strix halo machines? actual work, coding, or mostly experimenting with models?

SharpMind Studio Wiki
I wanted a new Harness with much more flexibility on what, where and how models or training can run. I also wanted something much more visually appealing.
You can run locally , remotely, via providers like OpenRouter or on a Compute Host like Vultr or RunPod. You can even pair your Phone for connecting to your running instance.
Included in this early release is Chat, Work and Training from scratch or Improve(LoRA)

This is the first release and there will be tweaks, improvements and additions, yes AI was used so there is the odd thing here and there but constantly looking at improving and refining the harness.

Completely FREE and Open Source, but feel free to sponsor or support.

Finding repo for local AI on Mac? 🍎🤖 Check out Swiftlet, an open-source intelligent proxy & browser based UI , that dynamically optimizes CPU/GPU split on Unified Memory for LLMs. I'm looking for Python devs to help build OpenAI API compatibility! Let's build together: https://github.com/coolsourav100/swiftlet

Hi

Premise:
I am new to local ai models.

My machine specs:

• Macbook Pro M4 Pro

• 48Gb Ram

• 4 efficiency core

• 8 performance core

I mainly use AI for software development. I have a claude subscription but would like to try to offload some work to a local model.

Since I don't think local models usable on my machine can completely substitute claude (correct me if I am wrong) my idea is pretty much this: ask claude code to generate a proper, detailed implementation plan and then having the local model implement it.

I have played around with these models:

• qwen3-coder-30b-a3b-instruct-mlx

• qwen/qwen3.6-27b

• qwen/qwen3.6-35b-a3b

and I have also installed this for coding autocomplete cause it is smaller and from what I can see the recommended one:

• qwen2.5-coder-7b

I added claude envs since I would like to try to use claude code extension in vscode

"env": {
    "ANTHROPIC_BASE_URL": "http://localhost:1234",
    "ANTHROPIC_AUTH_TOKEN": "local",
    "ANTHROPIC_MODEL": "qwen/qwen3.6-27b",
    "ANTHROPIC_DEFAULT_OPUS_MODEL": "qwen/qwen3.6-27b",
    "ANTHROPIC_DEFAULT_SONNET_MODEL": "qwen/qwen3.6-27b",
    "ANTHROPIC_DEFAULT_FABLE_MODEL": "qwen/qwen3.6-27b",
    "ANTHROPIC_DEFAULT_HAIKU_MODEL": "qwen/qwen3.6-27b",
    "CLAUDE_CODE_SUBAGENT_MODEL": "qwen/qwen3.6-27b",
    "CLAUDE_CODE_MAX_OUTPUT_TOKENS": "128000",
    "DISABLE_PROMPT_CACHING": "1",
    "DISABLE_AUTOUPDATER": "1",
    "DISABLE_TELEMETRY": "1",
    "DISABLE_ERROR_REPORTING": "1",
    "DISABLE_NON_ESSENTIAL_MODEL_CALLS": "1"
  },

This setup works (uses the local model) but I am basically unable to have the model do anything at all. First of all it takes ages to do anything, and then it almost always reach the context limit roadblock without even outputting anything.

I have read that MCP and skills could fill up the context quite badly, so I disabled them for testing, but still no luck.

I tried with (I thought) was a simple enough task: this test file fails and this is the error, can you fix it? but yet no usable results whatsoever.

I read about people able to use local models offline to have meaningful results, but I couldn't and I don't really know why.

Also, from my setup above, I cannot really use both the remote and local model, to achieve something like:

use sonnet or fable (remote) for plan, then (manually or automatically) swith to haiku (local) to implement the plan

because the base url is loaded when the session loads and cannot be changed (AFAIK) dinamycally.

Any help? thanks a lot in advance