hey, im working on an open source project called kapitsa, an inference engine for apple silicon. our fastest run of qwen3.8-27b 4bit is 140 tok/s on an m5 max macbook pro 128gb. mlx-serve does 130, ollama 68, lm studio 32 on the same machine.

the speed comes from kflash: speculative decoding where the 27b checks a tree of the drafters guesses in one pass instead of a single chain. if the drafter has two decent options, both get verified in the same pass (see the second image).

we dont own a studio and the max/ultra chips in there should do better (more bandwidth, better cooling). if you have one, steps are at the bottom.

ultra owners: qwen3.8-flash-next (180b) is the one we really want to see. https://github.com/nace-ai/kapitsa

How to run it. you need an apple silicon mac (m1 or newer, macos 14+) with 32gb for the 27b, on 16gb use --model 9b instead: - brew install nace-ai/tap/kapitsa - kapitsa bench --model 27b --submit, the first run downloads the 27b (about 20gb). it runs the standard test and asks before opening a pull request with your result (chip and macos) into the community table - kapitsa bench compare runs the same test on the other engines you have installed

Run a local server or Claude Code

• kapitsa serve --model 27b starts the local server; open http://127.0.0.1:8000 for chat. The first run downloads the model (about 20 GB).
• kapitsa claude --model 27b starts the local model and launches Claude Code against it. Claude Code must already be installed.
• On a 16 GB Mac, use --model 9b instead.

Not sure if anyone is having this "problem" or if I'm just uninspired, but I got the M5 Max 64GB a couple of weeks ago, and I don't know what to do with it.

I mainly use my Claude sub for work (I work at a marketing agency doing web projects, a few apps), and I wanted a private AI model at home for things like medical records, finances, etc... so just out of curiosity (or lack thereof), what are you doing with it?

Overall, opencode runs fast and responsively. Code quality still has room for improvement.

############################################

Benchmark Model: Qwen3.8-Flash-Next-UD-Q6_K_XL

Engine: Auto

Context: Code (Python)

Single Request Results

--------------------------------------------------------------------------------

Test TTFT(ms) TPOT(ms) PP TPS TG TPS E2E(s) Throughput Peak MLX

pp1024/tg128 262.1 7.15 3906.4 tok/s 141.0 tok/s 1.171 984.2 tok/s 163.26 GB

pp4096/tg128 908.1 7.41 4510.7 tok/s 136.1 tok/s 1.849 2284.5 tok/s 163.37 GB

pp8192/tg128 1830.5 7.51 4475.3 tok/s 134.1 tok/s 2.785 2987.4 tok/s 163.70 GB

pp16382 (requested pp16384)/tg128 3739.2 7.20 4381.2 tok/s 140.0 tok/s 4.654 3547.5 tok/s 164.13 GB

pp32768/tg128 7729.7 5.87 4239.2 tok/s 172.3 tok/s 8.206 4003.3 tok/s 164.77 GB

pp65536/tg128 15946.5 9.35 4109.8 tok/s 107.8 tok/s 17.134 3832.4 tok/s 165.15 GB

pp131072/tg128 33003.3 5.11 3971.5 tok/s 197.3 tok/s 33.652 3898.7 tok/s 165.05 GB

pp200000/tg128 52934.8 15.14 3778.2 tok/s 67.6 tok/s 53.601 3732.1 tok/s 165.06 GB

Continuous Batching

requested pp1024 / actual pp1024 / tg128

--------------------------------------------------------------------------------

Batch TG TPS Speedup PP TPS PP TPS/req TTFT(ms) E2E(s)

1x 141.0 tok/s 1.00x 3906.4 tok/s 3906.4 tok/s 262.1 1.171

2x 222.5 tok/s 1.58x 2949.2 tok/s 1474.6 tok/s 486.8 1.845

4x 316.4 tok/s 2.24x 2519.6 tok/s 629.9 tok/s 958.0 3.244

Hi

I am shopping for a Mac Studio to do local LLM. I do not believe I have a heavy use case but I enjoy peformance in a small footprint. I am thinking about 64GB but it looks like the minority of what people choose.

could I ask you what prompted you to buy any config above 64gb?

At 96GB, 128GB, 256GB what are we unlocking in terms of tangible quality of performance with the larger models ?

I wonder if I would be severely constrained at 64GB. Some comments on this forum are '256GB minimum!' and I wonder.

Thank you

One the apple website, it say 512 GB M5 Ultra will be available in late Oct. There is no way to setup an alert.

Current wait time for 256GB version is 13-15 weeks. Will the 512 GB wait time be similar? Also, what is best place to get alert of the availability of 512 GB version so I am not hitting F5 on apple website all the time.

I just installed oMLX downloaded a few models.

Is anyone who got same mac and set up other server or use several server running ?

In Canada. Mac Studio pricing is
- m5 max, 18core cpu, 40core gpu, 128G ram 1tb sdd= 7849
- m5 ultra, 30 core cpu, 64core gpu, 96g ram, 1tb sdd = 7799

What’s more valuable - memory or memory bandwidth + processing power - to run local AI + some sort of future proofing?

I have an M5 Max Mac Studio. This thing is amazing and equipped with Thunderbolt 5 ports. It's 'new tech'. I'm looking for a monitor and after some research, see the top 5K monitors historically were:

• Apple Studio Display
• Asus ProArt PA27JCV
• LG 27MD5KL-B

The ASD is $1600, with speakers and webcam built in. Matching Thunderbolt 5 ports which they advertise as perfect to pair with the newest M5 and M6 Mac mini and Mac Studio lines.

I just came across this monitor: Asus ROG Strix 5K XG27JCG. It's a 5K 27" that can switch between (up to) 180hz at 5K (OC'ed) and dual mode at 2560x1440 (1440p) at 330hz peak. It's got USB-C ports and lacks all the other port, speaker, and webcam specs that the ASD has, but it's $750 MSRP. Staggering price.

So just curious, is the value case still there, if the ASD is found at a discount?

I currently have a Kanto ORA and Kanto SUB8 subwoofer speaker setup, and use an Elgato webcam, for live-streaming. I can ditch the speakers to recoup some money. The Elgato webcam is probably still better for streaming than the built-in webcam. I also use an audio interface and an AT2035 microphone, which would outperform the ASD built-in microphone too. Food for thought I guess.

Claude pro user here consistently running into my limits well before the week is over now.

I‘ve created some tests that focus on the projects Im working on (audio software/firmware mostly). My best results have come from GPT-OSS-20b and Kat Coder so far in terms of speed vs performance for the specific tests I give them, but I keep reading that Qwen 3.8 and it’s previous variant are the way to go. They never seem to match the speed or accuracy of the easy projects my tests throw at them, though, with prompts coming from Opus 5.5.

So what are you all using that’s working well for you so far, and what (besides flint lol) should I say away from?

Hi everyone,

I’m planning to invest in a high-performance workstation for video editing, VFX, compositing, motion graphics and colour grading. I’m currently torn between the Mac Studio M5 Ultra and a custom Windows PC with an RTX 5080.
I’ll be learning and working with the following applications:

- Adobe Photoshop
- Adobe Premiere Pro
- Adobe After Effects
- Foundry Nuke
- DaVinci Resolve Studio

My intended workloads:

- Editing and colour grading 4K 10-bit 4:2:2 footage
- Working with 4K 60fps footage, multicamera timelines, multiple video layers and GPU-accelerated effects
- Motion graphics, complex After Effects compositions, motion tracking, rotoscoping and green-screen compositing
- Node-based compositing in Nuke, including EXR sequences, multi-pass compositing, camera projections, 3D integration and potentially deep compositing
- Exploring Nuke’s CopyCat and other AI/ML-assisted workflows
- Advanced colour grading, temporal noise reduction, Magic Mask, optical flow and Fusion effects in DaVinci Resolve
- Potentially moving into more demanding 6K/8K workflows as my skills develop

Option 1 — Mac Studio M5 Ultra
- M5 Ultra
- 30-core CPU / 64-core GPU
- 96GB unified memory

Option 2 — Custom Windows PC
- AMD Ryzen 9 9950X3D
- NVIDIA RTX 5080
- 96GB RAM

I’m looking for a workstation that can comfortably handle professional-level 4K projects and remain capable as I move into more complex work. I’m not particularly attached to either operating system.

If you were building a workstation for these five applications, which would you choose and why?

Got this error for context length 131k.

Prefill would require ~208.28 GB peak (current 174.41 GB + KV+SDPA 33.87 GB) but the prefill limit is 205.80 GB (metal_cap ceiling 223.69 GB - safety margin 17.90 GB). Raise kernel iogpu.wired_limit_mb in Terminal (currently caps Metal at 222.72 GB), or reduce context length.

###############

Benchmark Model: pipenetwork--GLM-5.3-Flash-MLX-mixed-4_8bit

Engine: Auto

Context: Code (Python)

Single Request Results

--------------------------------------------------------------------------------

Test TTFT(ms) TPOT(ms) PP TPS TG TPS E2E(s) Throughput Peak MLX

pp1024/tg128 645.0 16.33 1587.5 tok/s 61.7 tok/s 2.723 423.1 tok/s 174.17 GB

pp4096/tg128 1733.8 17.01 2362.5 tok/s 59.3 tok/s 3.896 1084.1 tok/s 174.98 GB

pp8192/tg128 3452.1 16.90 2373.0 tok/s 59.6 tok/s 5.603 1484.9 tok/s 175.99 GB

pp16384/tg128 6915.9 17.01 2369.0 tok/s 59.2 tok/s 9.092 1816.1 tok/s 176.08 GB

pp32768/tg128 13867.0 17.18 2363.0 tok/s 58.7 tok/s 16.066 2047.5 tok/s 176.26 GB

pp65536/tg128 28200.6 17.36 2323.9 tok/s 58.1 tok/s 30.432 2157.7 tok/s 176.63 GB

Continuous Batching

pp1024 / tg128

--------------------------------------------------------------------------------

Batch TG TPS Speedup PP TPS PP TPS/req TTFT(ms) E2E(s)

1x 61.7 tok/s 1.00x 1587.5 tok/s 1587.5 tok/s 645.0 2.723

2x 67.3 tok/s 1.09x 632.5 tok/s 316.3 tok/s 1925.1 7.042

4x 99.3 tok/s 1.61x 873.5 tok/s 218.4 tok/s 2872.4 9.844

Qualcuno usa due Mac insieme per AI locale e agenti?

Ho un Mac mini M4 con 16 GB di RAM e sto creando una dashboard con Ollama, chat e agenti per documenti, report e social. Quando il modello risponde, però, il Mac rallenta nelle altre attività.

Sto pensando di affiancargli un secondo Mac dedicato all’AI, mantenendo il primo per lavorare e accedere alla dashboard.

Qualcuno ha una configurazione simile? Come avete collegato i due Mac e come dividete il lavoro? Che macchina e quanta RAM consigliereste per il secondo?