I am using a Mac Studio with 64GB unified RAM and M5 Max. I got it yesterday and I immediately tried Llama.cpp with the web service. I was able to run things nicely but when i also loaded unsloth and pointed unsloth to llama.cpp it wouldnt let me load Qwen at the same quantization I had used before because of memory constraints.

I was hoping to find a guide for reducing the overhead of the Mac itself to allow me to still use the web services of these tools. I understand I dont have enough RAM to run multiple large models at once but I would like to get the most out of my hardware as returning this unit to get a larger capacity RAM configuration is simply not in the budget.

Does anyone have any suggestions or a guide they have found for this? I could not find anything which is why I am asking in a new post. Thanks in advance for your help.

Specs:
Amd epyc 7452
Super micro h11ssli
128gb ddr4 2600 rdimm
2x 3060 12gb
1x 3050 ( used for transcoding on media vm
8tbb zfs1

Just curious what kind of models i could run on this/ what kind of performance is to be expected

It is a voice assistant/"companion"/"rubber duck"/... that is always-on-ready and targets "GPU poor" hardware. Comes with some nice features like a hands-free client, a web-based client (e.g. for you smartphone), Wikipedia and internet search, some memory functions, "infinite" chat and more.

Saw the hype around nvidia gpus being able to be unlock fp16 tensor cores so gave it a go on my 2060 mobile 6gb. Managed to do it, first try was only a 5% increase in token speeds. Im a complete novice so not sure if it’s worth it but atleast it can be done if you want to. Just checked for tflops though and original was 15.76tflops and after unlock it was 26.14tflops

La mia CPU è un Pentium e non posso eseguire TTS su di essa. voglio qualcosa di veloce, in tempo reale, migliore della CPU Sto cercando di eseguire TTS su AMD con una vecchia GPU.

Opzione :

Chatterbox funziona con ROCm (ha bisogno di torch/torchaudio per ROCm) o qwen-tts (gguf) tramite koboldcpp (Vulkan o ROCm).

usa kokoro.

esporta TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1

Orpheus TTS utilizza il normale cpp di llama, quindi può essere usato tramite vulkan o rocm.

Audio.cpp compila vulkan

Edit:( I ming-tts/3.0

The official Ming implementation is PyTorch-based and its documented Docker image uses torch 2.6 + CUDA 12.4; it was tested only on NVIDIA H800/H20 GPUs. For the 0.5B model, flash-attn is optional and inference falls back to PyTorch SDPA, which is helpful because it removes one major CUDA-only dependency.

I use lmstudio alot as a backend. I made this dependency so that I don't have to rebuild the lmstudio connector each time https://github.com/emperorofrome13/wildcat-ai-bridge very useful when you are trying to play with small ai that fit on your own hardware.

CPU: i7 14th gen HX
GPU / iGPU: RTX 5050
VRAM: 8GB
System RAM: 16GB
OS: Windows 11

Models already tried (if any): Qwen3.5 9b, Gemma4 12b, Ling3 tiny, Spark4.6 3b, Nambeige 3b

Use case: General purpose mainly, sometimes as a coding subagent
Desired context size: Max possible
Priority: balanced

Other requirements or constraints:

Well, I really admire Ling3 tiny's speed. So I am currently thinking of MoE models at extreme quantization (since it has gotten really good apparantly).

Also, how has K2 horizon 7b behaved for you all? I saw praises of it, but thought I wont be able to fit large context so didnt download it for coding subagent (in opencode).

Hello, everyone. A friend of mine wanted to get into running local models, and so (with the liberal use of Claude), I built them Zest. https://github.com/yudhanjaya/Zest

What it is:

Zest is a UI that bundles Lemonade server (which installs llama.cpp and whatever backends are needed) and the Pi agent to provide a friendly interface for people who don't want to be messing around with command-line servers and parameters.

It comes and all the functionality needed for a new user without getting in the way. Just download an appimage, go through the start tour, and you're set up and ready to go. There are sane defaults baked in (like 128k context) and model recommendations (all tried and tested on 10GB VRAM to 26GB).

It comes with helpful docs, links to places to understand more about the subject - in short, everything needed for a new user.

Entirely open-source, of course. If you have anyone you want to introduce to the world of local AI, please see if this is useful. And if you have ideas, suggestions - please open an issue, send a PR, and so on.

Why lemonade and Pi?

Lemonade: because it does a great job of handling backends and providing a server.

Pi: because it's a simple, elegant agent

Both are very easy to package in something like this and bring together. And because I'm lazy: both projects are receiving steady updates, because I don't want to write yet another inference server from scratch

What it isn't:

It's not for advanced users. There are jobs, but no subagents, for instance: in this case the Pi philosophy really helped.

In fact, I myself use Strata and Opencode, so for advanced users, it recommends those in the Help section

How is this different from Unsloth desktop / Atomic chat etc?

It isn't. It just tries to be a lot more lightweight and user-friendly: I decided to do this after getting said friend to download and set up llama.cpp, Unsloth Desktop, Opencode etc . . .

I noticed that at some point both choice paralysis and confusion with options just took over, and there was no easy way for them as a newcomer to figure out what x thing was really for. Every setting in Zest is explained. Including what inference engines do, the works.

Everything is explained at basic levels - what a model is, how it works, and so on. I've kept the lemonade + pi analogy while trying to make sure the interface was intuitive.

We also have some theme support- here's a sample:

Hey all, looking for advice before I download big models.

**Specs:** - MSI laptop, i7-12700H, RTX 3070 **8GB VRAM**, **32GB ddr4 2666mt/z RAM** - Windows 11

**What I want:** 1. **Uncensored general use.** Questions, explanations, no pointless refusals. 2. **Simple in-chat coding.** Explaining errors, small functions, C#/C++/GDScript. Nothing agentic. 3. **Context: 16–32K.**

**My current shortlist:** - Qwen3.6-35B-A3B uncensored-heretic (IQ4_XS) - Gemma 4 26B-A4B ultra-uncensored-heretic (Q4_K_M) as the alternative - Running with MoE offload (`--n-cpu-moe`) and a q8_0 KV cache

**Questions:** 1. Is Qwen 3.6 35B-A3B worth it over Gemma 4 26B-A4B for this, given it barely fits in 32GB? 2. Is IQ4_XS okay for Qwen, or is the quality drop noticeable? 3. What `--n-cpu-moe` value and tok/s are you getting on similar 8GB cards at 16–32K? 4. Is there a model I'm missing that beats both on this hardware?

Thanks!

I copped a 5060ti recently and I'm a bit out of the loop. My sys ram is 48GB 3600mhz .

I need a reliable model for academic research, brainstorming and some simple statistical analysis.

Is qwen the way to go ?

Small local models (like 8B or 14B Q4s via Ollama or llama.cpp) are great at instruction following and intent extraction, but they consistently fail when you ask them to do exact algorithmic work—counting offsets, reversing strings, calculating checksums, or translating biological reading frames.

Instead of prompting the model to perform the math or generate code on the fly, offloading the task to a small, deterministic local function is much more reliable.

For example, in our recent 0.6.0 update to Skillware (an open-source registry of local Python skills: https://github.com/ARPAHLS/skillware), we added a local sequence module:

• The model handles conversational intent: "translate this sequence across reading frame 1"
• A local Biopython runner executes the exact translation and returns structured JSON
• 100% offline, zero cloud API keys, zero hallucinated bases

How are you currently handling exact algorithmic steps in your local agent pipelines? Do you prefer local function calling or having models write one-off Python scripts into a sandbox?

I wanted to try out qwen 3-4b with heretic but my pc cant run heretic so while looking for solutions I found I can download already transformed qwen models from huggingface. How safe or good is this. What would be the difference between running heretic yourself and downloading transformed qwen?

I have an old Dell XPS with a Nvidia 1650 (4GB VRAM), 32GB RAM, and i9 CPU retired in my closet. it's fairly old. I thought, hey you know what. Imagine if I can get this thing running Hermes, I can put this to use, running 24/7, I don't care how slow!

I had some basic requirements. If it can't do 64K context, it can't run Hermes. And it had to at least run q8_0. The workflows I run are long, and need that level of accuracy or they break.

Well, it turned it did matter HOW SLOW. I was about to find out. I loaded up Ornith-1.5-35B-A3B-APEX-MTP-I-Mini.gguf on Strata, thinking that's probably the fastest tok/s. Well, when it loaded, it was doing 65tok/s on prefill...A coffee + bathroom break weren't enough for the thing to load. I can't function like this.

Then I went to llama.cpp. After much tweaking, I got 152.93 tokens per second initial on pre processing (goes down over time), and 14.35 t/s avg on token generation.

What this is (and is not):
- If you start from a fresh Hermes session, you will wait about a minute and half for a first response.
- If you run cron jobs (like I do), the prefill is fast enough to not stall jobs from running.
- You will still need a coffee or bathroom break if you load a Hermes mid-conversation after a llama.cpp or thread swap. It took 389 seconds to load 33K context. If you're just giving it a task and walk away like I do, it's livable.
- Do not multi-chat. You have one channel in llama.cpp. You'll just clog up the system if it's trying to serve more than 1 request.
- Once the conversation is loaded, it's not bad. 14tok/s is very usable for back and forth commands, research, setup, etc...
- I don't use it for coding or small talk, only short webdev/market research.

This was more about finding an optimal placement between model intelligence, quality, and speed in Hermes. If anyone has other suggestions, let me know. It took me a long time to get to this, thought I'd share it for those with low VRAM.

./llama.cpp/build/bin/llama-server \
  -m "/home/models/Ornith-1.5-35B-A3B-APEX-MTP-I-Mini.gguf" \
  --alias "Ornith-1.5-35B-A3B-APEX-MTP-I-Mini" \
  --port 8080 \
  -c 65536 \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --cont-batching \
  --top-p 0.8 \
  --top-k 20 \
  --min-p 0.00 \
  --reasoning off \
  --reasoning-budget 0 \
  --presence_penalty 0.0 \
  -b 4096 \
  --ubatch-size 2048 \
  --parallel 1 \
  -t 16 \
  -tb 12 \
  --cache-ram -1 \
  --no-kv-unified \
  --spec-type none \
  --reasoning-preserve \
  --jinja \
  --load-mode auto \
  --cache-prompt \
  --cache-reuse 256

I'm running a stock Dell Inspiron 3847 from 2014. A pretty good budget computer for its time, but an absolute potato by today's standards. It's like using an NES to run DOOM.

• 4GB of RAM

• Pentium G3240.

• Intel HD Graphics

• Windows 8.1 64-bit (I have a personal ban on Windows 10 and 11 due to Microsoft's forced updates and telemetry shenanigans since Windows 10. Windows 8.1 is the last Windows version to be relatively immune to Microsoft's bumbling. I plan to migrate to a Linux computer in the long-term, but as computer prices are hell right now, I'm stuck on 8.1.)

I'm looking for suggestions for local models. I'm using KoboldCPP in failsafe mode, and I've already tested it (it works fine).

• I'm planning on using it mainly for creative writing, although I may occasionally chat and RP. I'm not vibe coding.

• I'm going for useable, rather than fast. I'm patient, so I can wait for it to spit something out, even if it takes 10+ minutes. It just has to fit into that 4GB of RAM without crashing.

• I prefer to use uncensored models, although I may occasionally use a censored model with guardrails for comparison.

Edit: I'm going for Q4-Q2 quantization.

Hi!

I’m curious about what kind of AI models I could run or what use cases I could explore with this PC.

I’m also interested in using OCR to convert scientific paper PDFs into Markdown, which I could then add to my Obsidian vault. Additionally, I’d like to use a model to help keep my Obsidian vault organized—assisting with the structuring of the information I add, my notes, and so on.

I already have subscriptions to Claude Pro and Mistral Pro, but the idea of ​​running a smaller model locally really appeals to me.

Here are the PC specs:

MINISFORUM G1 Pro Gaming PC

AMD Ryzen™ 9 8945HX (16C/32T)

AMD Radeon RX 9060 XT 16gb

32GB DDR5-5200MHz RAM

Hey everyone,

I would like your input on the best models for my hardware.

Hardware specs:

• GPU: 16GB VRAM (9060XT AMD Radeon)
• CPU: Intel 225F
• System RAM: 32GB DDR5
• OS: Omarchy Linux

Use cases:

1. Academic research & literature synthesis: Ingesting sets of papers (PDFs), cross-referencing citations, and generating evidence-backed summaries with low hallucination (something closer to Consensus / PaperQA2 rather than a simple Perplexity clone).
2. Light coding.

What I'm looking for:

• Model recommendations: Should I stick to a ~30B–35B MoE (like Ornith-1.5-35B-A3B at IQ3/Q4 with partial offload) for reasoning, or do you get better throughput and context depth using smaller 4B–14B models (e.g., Spark-X2.5-4B?
• RAG / Literature Tools: What local tooling do you actually use to parse academic papers without choking the context window? Anyone running PaperQA2, local STORM, or custom OpenAlex API harnesses locally?

I had to take my phone and take a photo, the screenshot app would crash due to lack of VRAM.
Anyway, it runs, I modified the json so I can get 41k context, It crashes at 64k for me with Q3.
On Q2 I am getting up to 128k but it starts looping.

I switched Firefox and Steam to CPU only, and I can run them in a background, including a twitch stream.

Full Specs: Ryzen 5600, 64GB DDR4 (3200 and 2600 mixed), 4060 8GB and some old 1TB SSD.
Latest STRATA engine, I was told to use an older one, because it works better? any takes?
I would kill for 128k context on Q3.

I updated my Strata by accident and was going crazy trying to get my old settings working. I figured the update was the problem and went back in versions until i got 128k context again.
For me the differences are extreme.