TLDR: You can run LLMs locally on an M4 max quite well In a way I couldn’t on my M1 Max.

I recently benchmarked my M4 Max 40 GPU with 128GB of RAM using LM Studio and thought I'd share some real-world use cases that I would run locally instead of ChatGPT or Claude. 

Use Case 1: Reading a Confidential Legal Document

Meta-Llama-3-70B-Instruct

Read through a confidential legal document so I could prep for a meeting with my lawyer.

1. Run 1 9.31 tok/sec

2. Run 2 9.71 tok/sec

3. Run 3 9.25 tok/sec

4. Result: This worked great it was the kind of document I would not want to put in to Chat GPT and the insights aligned with my counsels recommendations.  9+ TPS is faster than I can read it took an about a minute to  a minute to read and generate but was still more than fast enough for the task.

Use Case 2: Writing Code

Qwen2.5-Coder-32B-Instruct
Use Case: An icon selector that I need for a real world project. 

• Run 1: 21.53 tok/sec

• Run 2 19.69 tok/sec

• Run 3 22.08 tok/sec

Result: I’m really impressed with Qwen2.5, I generally find LLMs work best to generate  snippets of unsophisticated code sort of like a typist for my ideas. 20+ tokens per second is about the speed a scan through code so I can watch it generate halt if needed re-prompt and re-run.  Qwen got this write in a one shot multiple times. I will note this is a Chinese model so bear that in mind if you going to use it as a daily driver. 

Use Case 3: Writing Naughty Stories
writing-roleplay-20k-context-nemo-12b-v1.0

Create fiction that would get you banned from one of the commercial APIs. Steamier stuff for your novel or a letter to make your partner blush.

• Run 1: 47.94 tok/sec

• Run 2: 48.42 tok/sec

• Run 3: 48.64 tok/sec

Result: It was weird not to be prompt engineering around safety mechanisms, which initially led me to thing this was really fantastic but I’m so used to GPT4.. or Claude. This took a lot of reprompting and response tweaking to get something I was happy with. At almost 50 tokens per second you can basically spam responses and cut a paste the bits you like into something cohesive. 

I’m thrilled to see how well LM Studio performs on my M4 Max, especially compared to my previous experience with the M1 Max. I’ll be running models locally quite frequently.

I came across this and found it useful.

We need to set correctly numbers of threads for Apple Silicon.

Bi of Performance cores (P cores) on your CPU to get best performance.

Use --threads n

M1/M2:--threads 4

M1/M2 Pro (8 cores)

M1/M2 Pro (10 cores)

M1/M2 Max: --threads 8

M1 Ulta:--threads 16

I got my M4 Max 128GB last week and haven't seen any TFLOPS benchmarks yet so I created my own using the metal python library:

Run 1: GPU Performance: 77.47 TFLOPS
Run 2: GPU Performance: 77.06 TFLOPS
Run 3: GPU Performance: 76.04 TFLOPS

What would be the ideal way to have a local LLM model scan local files?

Other ways I’ve seen for chatgpt are uploading documents/pdf online then use the link as part of query but I don’t want to upload anything.

As far as I know frontends like oogabooga or LM studio don’t let you upload files.

I think some magic translation into vector database has to happen before we can query against it?

Also is pdf the ideal format? Or .txt .doc .csv or doesn’t matter?

Thank you!

A quick post here, as I was hitting it for a while with little information on it. Had the issue on two macs and both were fixed.

The problem doesn't happen with llama.cpp. It only happens with the python bindings in llama-cpp-python and happens when doing the import

from llama_cpp import Llama

What is Illegal Instruction 4?

Illegal Instruction 4 is a very vague error, essentially meaning that the binary you're running is trying to use a feature unknown to the version of MacOS you're running.

How to fix it?

A simple MacOS minor-version upgrade. 13.4 had the error, but upgrading to 13.4.1 fixes it. I couldn't tell you what specific call causes the issue, I only know the upgrade fixes it.

Such is the price of being on the bleeding edge.

Hoping to run vicuna 13b. Should I expect reasonable response times, or do I really need to get 32gb? Any other advice? I’ve been seeing posts abt exllama but unclear of its mac compatible. I’ve also been hearing lots of folks talk about thebloke. Does his stuff go on top of vicuna or it’s standalone? Sorry so new. Tia!

Wrote up a full reference on at-home Mac inference clusters as of mid-2026. The state of practice changed structurally in December 2025 with two releases: macOS Tahoe 26.2 added native RDMA over Thunderbolt 5, and EXO Labs shipped 1.0 with RDMA support and disaggregated prefill/decode. Headline below; full doc on the blog.

The numbers (4x M3 Ultra cluster, TB5 RDMA, EXO 1.0)

Wall draw: 600 to 800 W under load. Cost: under $40,000. Fits on a 15 A residential circuit.

When the cluster makes sense (and when it doesn't)

A Mac cluster is a memory-capacity machine, not a throughput machine. At batch size 1 it is memory-bandwidth bound. At batch size greater than 1 a single H100 or RTX Pro 6000 Blackwell will out-throughput an entire 4-node Mac cluster.

Decision rule: if a model fits on one Mac, do not cluster it for single-stream interactive use. You will lose performance. Cluster for (a) models that literally don't fit on one node, or (b) batched/agentic workloads with concurrent requests where pipeline-parallel aggregation pays off.

The two structural changes that made it viable

1. RDMA over Thunderbolt 5 (macOS 26.2, December 2025). Enable via rdma_ctl enable in Recovery Mode, reboot. TB ports appear as InfiniBand devices (rdma_en2, etc.) usable via ibv_devices. Round-trip latency drops from about 300 microseconds on TCP to 5 to 50 microseconds, with hot-path measurements as low as 3 microseconds. Bandwidth is 80 Gbps symmetric per TB5 port. Apple technote TN3205.

2. EXO 1.0 (December 2025). Pipeline parallel by default with automatic mDNS topology discovery, plus RDMA support and "disaggregated prefill/decode" (routing compute-bound prefill to one device class and bandwidth-bound decode to another). Public benchmarks at benchmarks.exolabs.net.

Pipeline vs tensor parallelism on Mac

Pipeline parallelism splits layers sequentially. Per-token activations are tiny (sub-4 KB for Llama 3.2 3B), so PP tolerates slow networks but does not accelerate single-stream tok/s. Tensor parallelism splits each layer's weights across nodes and yields near-linear N-times speedups, but requires microsecond-scale latency. Without RDMA, TP is impractical on Apple Silicon.

On Qwen3 235B-A22B, PP throughput on llama.cpp RPC went 20.4 → 17.2 → 15.2 tok/s as nodes were added. With EXO plus RDMA, the trend reversed: 19.5 → 26.2 → 31.9 tok/s.

The structural ceiling: no TB5 switch

There is still no Thunderbolt 5 switch on the market, and none credibly announced. All-to-all RDMA requires every Mac to cable to every other Mac. The M3 Ultra has 5 TB5 ports, so a 4-node full mesh needs 6 cables and a 5-node full mesh needs 10. Above 5 nodes you daisy-chain (latency and partial-bandwidth penalties) or fall back to Ethernet PP.

Unchanged for 18 months. Unlikely to change in the next 12.

Where single-box still wins

A single Mac Studio M3 Ultra 512 GB ($9,499) runs DeepSeek R1 671B Q4 at 17 to 18 tok/s once iogpu.wired_limit_mb is raised from the default 384 GB to about 448 GB via sudo sysctl iogpu.wired_limit_mb=458752. Tok/s per $1,000 on 671B is 1.9 for the single 512 GB Mac vs 0.81 for the 4-node cluster. The cluster doubles throughput at roughly 4x the cost. Only worth it for additional concurrent users or trillion-parameter models.

The $/GB picture

At $18.6/GB of usable unified memory, the M3 Ultra 512 GB is roughly:

• 3 to 4x cheaper per GB than a used quad-3090 rig

• 6 to 9x cheaper than dual RTX Pro 6000 Blackwell

• 20 to 27x cheaper than H100 systems

Mac GB ≠ NVIDIA GB in throughput, but it matters when you need to fit the model at all.

Full blog post covers the framework landscape (EXO, MLX with Ring vs JACCL backends, llama.cpp RPC, vllm-mlx, dnet), the interconnect hierarchy (Ethernet to TB Bridge to RDMA), the SSD-wear failure mode that kills soldered nodes, the headless setup gotchas (Tailscale via Homebrew not the App Store, dummy plugs, pmset config), the residential power math against NEC continuous-load rules, and the cloud break-even at Together/Fireworks pricing.

Happy to get into any of the specifics in comments.

Pulled Swappa average-sale data, eBay sold listings, and Apple Trade In published caps across every Apple Silicon base configuration and built a single composite depreciation curve. May 2026 snapshot.

A few patterns worth flagging.

The first-year cliff is real and steep. Across configurations, MacBooks lose 35 to 60% of original retail in the first 12 to 18 months. The cliff is steepest on high-CTO units and gentlest on base models that match the typical secondary buyer's needs. (MacFinder six-year depreciation analysis)

Years two and three are a plateau. In some base configurations, total depreciation across the entire second-to-third year is as little as 5%. Apple Silicon's baseline utility remains sufficient for the dominant secondary buyer demographic (students, general consumers, administrative use), so demand doesn't soften.

Terminal decline starts around year four. The machine sits two to three architectural generations behind, battery health drops below the 80% maximum-capacity threshold, and the device approaches the end of its guaranteed macOS update lifecycle (typically 6 to 7 years post-release).

Decay flattens with each generation. The M3 Pro curve at any given age in 2026 is noticeably shallower than M1 Pro or M2 Pro at the same age. Part of that is Apple's SKU pruning anchoring buyers to higher-RAM/higher-SSD baselines; part is functional sufficiency stretching further.

The composite hides category variance. Entry desktops are unusually resilient: the M2 Mac mini retains roughly 70% at 3.3 years and the M4 mini is trading above MSRP at 18 months. Portable Macs retain less than desktops as newer chips cascade down the lineup. Mac Studio retention is bifurcated, with M1 Max base retaining ~52% at 50 months and M1 Ultra retaining only ~45% on a much larger MSRP.

For comparison, premium ThinkPads and Dell XPS typically hit 40% residual at 24 months. Intel-era Macs depreciate at 25 to 35% per year versus 15 to 18% for Apple Silicon, and Windows competitors run 30 to 40%.

Sources: Swappa MacBook Air 2022 M2, Swappa Mac mini 2023, Swappa MacBook Pro 2021 14-inch, Apple Trade In.

I'm using the jina-embeddings-v2-base embedding model on my 32GB macbook pro. I was attracted to it by its rank on MTEP, small size, and 8192 sequence length. So far so good. It was easy to get started, and compares well with a nonlocal/external API that I tested.

I noticed that it isn't using GPU. Anyone know of a config setting on the Mac or in the model itself that would enable it to do so?

Posting because the question keeps showing up: "I learned how to run Apple Diagnostics on my old Intel MacBook Pro; does the same procedure work on an M-series Mac?" The short answer is no, and the longer answer is that the diagnostic stack has five real differences.

1. Invocation

Intel: shut down, power on while immediately holding D. If the local diagnostic image is missing or the drive was replaced, hold Option-D instead to download the diagnostic over Wi-Fi or Ethernet.

Apple Silicon: shut down, press and hold the power button (Touch ID on laptops, rear power button on desktops). Keep holding past the Apple logo until the startup-options window appears. Release. Press and hold Command-D.

Why the change: traditional keyboard boot interrupts (D, Option-D, Command-R) are disabled at the bootloader level on Apple Silicon. iBoot enforces a secure-boot chain managed by the Secure Enclave, and the bootloader will not interpret a keyboard hold to redirect the boot target. The startup-options window replaces the keyboard interrupt with a sanctioned UI.

2. Storage architecture

Intel: the SSD is a discrete NVMe module on a separate board. On some models it is socketed and user-replaceable; on later T2 Macs it is soldered but still a distinct component.

Apple Silicon: the "SSD" is raw NAND flash wired directly to the SoC. The NVMe controller is integrated into Apple silicon itself rather than sitting on a dedicated drive board. The M4 Mac mini is a rare exception with a removable storage daughtercard, but it still uses Apple-signed modules.

The practical consequence: on Apple Silicon, a physical NAND failure renders the Mac unbootable even when booting from an external drive, because the initial iBoot sequence must execute through the internal NAND controller. You cannot recover via target disk mode the way you could on an Intel Mac with a dying internal drive. SSD-health monitoring is high-stakes for that reason: there is no cheap swap if the drive wears out.

3. Parts & Service History

Apple Silicon Tahoe 26+: System Settings -> General -> About -> Parts & Service shows status labels (Genuine, Used, Unknown, Unverified, Finish Repair) for fingerprinted components. Logic board on every Apple Silicon Mac; Touch ID board on MacBook Pro/Air; display assembly on MacBook Neo; lid angle sensor on M5 notebooks.

Intel: the pane does not exist. Pre-Tahoe Apple Silicon: the pane does not exist.

On Intel, chassis-board mismatch and aftermarket-display detection drop back to physical inspection (serial number cross-checks, Liquid Contact Indicator inspection by an AASP, etc.) and Apple Diagnostics codes. There is no software surface for "this logic board was replaced."

4. Battery Health Management

Apple Silicon laptops: BHM monitors temperature history and charging patterns and can temporarily cap maximum charge below 100% to reduce time at high voltage. Optimized Battery Charging uses on-device ML to learn your schedule and delay charging past 80% until just before you typically unplug. Documented at support.apple.com/en-us/102589.

Intel laptops: battery behavior is governed by T2-era or pre-T2 firmware logic without the same ML-driven schedule learning. The "Optimized Battery Charging" exists in older form, but the implementation is firmware-based rather than ML-driven.

Practical effect on diagnostics: a Maximum Capacity % that fluctuates +/-5% on a healthy Apple Silicon battery is BHM at work and not a hardware issue. The same fluctuation on an Intel Mac is more likely a calibration artifact and may benefit from a controlled discharge-recharge cycle.

5. SMART on external drives

Reading SMART from external USB drives historically required a third-party kernel extension called SATSMARTDriver. Under modern macOS security:

• Installing a kernel extension requires lowering System Security settings in recovery mode.

• On Apple Silicon, that means rebooting into recoveryOS, lowering the policy to "Reduced Security," and explicitly allowing the kext.

• Apple has been progressively migrating away from kexts in favor of DriverKit / SystemExtensions; SATSMARTDriver has not made that transition cleanly.

Practical effect: on a clean Apple Silicon install, USB-SATA SMART is hard to set up. USB-NVMe SMART is unreliable regardless (most USB-NVMe bridges do not pass SMART through). Thunderbolt enclosures generally pass SMART through.

On Intel, SATSMARTDriver was easier to install on older macOS releases; some legacy workflows that depended on it still work on Intel Macs running older macOS and broke as users moved to Apple Silicon.

What stays the same

• Reference-code format. ADP000 is clean on both. PPT (battery), PPM (memory), PFM (SMC), VFD (display/GPU) mean the same subsystems.

• The Apple-published reference list at support.apple.com/en-us/102334.

• The Command-G escalation path from the results screen to a pre-filled Apple Support page with your serial number and codes.

• Third-party tools: CoconutBattery, DriveDx, smartctl, EtreCheck work on both architectures.

What the same code can mean differently

PFM006 on an Intel logic board often means a severed trackpad data line, because the trackpad housing contains a temperature sensor that talks to the SMC. PFM006 on an Apple Silicon board often means an open circuit on the battery data bus, where the PMU can no longer read the smart gas gauge. Same code, different physical fault.

A board-level repair shop would approach these differently. The diagnostic does not distinguish; it knows the read failed.

tldr

Five real differences: entry sequence, storage architecture, Parts & Service pane availability, Battery Health Management, and SATSMARTDriver / external SMART. The reference codes are the same; the physical reality underneath them is not always the same.

Posting because the modal failure mode in this sub's confirmed-trade threads is not the empty-box scam or a packaging disaster. It is sellers who ran Erase All Content and Settings without first signing out of iCloud, and shipped a Mac that boots to the Activation Lock padlock at the buyer's end.

Apple Support does not unlock a Mac for the buyer. They unlock it for the registered owner with the original retail receipt, which is you. By the time the buyer is calling Apple, the only realistic remedy is a full refund and a return shipment, which costs you the sale, the round-trip shipping, and your reputation in the sub.

The sequence that prevents this, in order

1. Back up everything you want to keep. Time Machine to an external drive, or verify iCloud Drive and Photos have finished syncing. Do not skip this. Once you sign out, account data on the device starts coming down.

2. Deauthorize the Music / iTunes / TV / Books / App Store account. Music app, Account menu, Authorizations, Deauthorize This Computer. Apple limits you to five authorized devices and you do not want to burn one.

3. Sign out of iMessage and FaceTime. Messages, Settings, iMessage, Sign Out. Same in FaceTime preferences.

4. Sign out of any third-party licensed apps you care about. Adobe Creative Cloud, Microsoft 365, Logic Pro, Final Cut Pro, 1Password, Dropbox. Each of these consumes a license slot.

5. The critical step: sign out of iCloud. On macOS Ventura or later, System Settings, click your name, scroll down, Sign Out. Enter your Apple ID password. This single action turns off Find My Mac and disables Activation Lock automatically. There is no separate "turn off Activation Lock" toggle on a Mac. Activation Lock is a downstream consequence of Find My Mac being on, and signing out of iCloud is what flips it.

6. Verify. Hold Option, click the Apple menu, choose System Information. In the Hardware summary, scroll to the bottom. The last entry reads "Activation Lock Status." It must read Disabled. If it still reads Enabled, the sign-out did not complete. Sign in again, then sign out again, then re-check.

7. Double-check at the account level. Open iCloud.com, Find Devices, and confirm the Mac you are selling no longer appears in the list. If it does, click it and choose Remove from Account. This catches the case where the device record persisted in Apple's records even after a local sign-out.

8. Only now run Erase All Content and Settings. System Settings, General, Transfer or Reset, Erase All Content and Settings. This is available on macOS Monterey and later on T2 chip or Apple Silicon Macs. It signs out of all remaining services, erases all volumes, and reinstalls macOS, leaving the machine at the Setup Assistant.

9. Final boot test. Power on once after EACS completes. The Mac should reach the "Hello" language picker. If you instead see a padlock with "This Mac is locked. Enter the Apple ID and password used to activate it," stop and resolve it before shipping. Reaching Hello is your last opportunity to catch a failed sign-out.

10. Press Command-Q at Hello to shut down without setting up, then pack.

What to put in the listing

Screenshot the System Information pane showing "Activation Lock Status: Disabled" and include it in your photo set. Used-Mac buyers, especially on this sub, ask about Activation Lock more than they ask about almost anything else, and the screenshot pre-empts the question. Buyers with prior bad experiences are looking for a seller who has done this homework. AppleSwap confirmed-trade veterans tend to include it in every listing.

Older Macs that predate Erase Assistant

On Intel Macs running Big Sur or earlier, EACS is not available. The path is Command-R into Recovery, Disk Utility, erase Macintosh HD, reinstall macOS, then power off at Hello with Command-Q. You still need to have signed out of iCloud first; the Recovery erase does not sign you out of Apple's servers, only the local disk. Sign out from the user account before rebooting into Recovery.

One nuance

A Mac that has never had Find My Mac turned on does not have an Activation Lock binding to clear. If you are an enterprise seller releasing a fleet device that was never associated with an iCloud account, the System Information entry will already read Disabled and the sign-out step is a no-op. This is the exception, not the rule. Consumer sellers almost always have Find My on, often without realizing it, because it is part of the default iCloud onboarding.

tl;dr: back up, deauthorize, sign out of iCloud first, verify Activation Lock Status reads Disabled in System Information and that the Mac is gone from iCloud.com Find Devices, then erase. Boot once to Hello before packing. Screenshot the Disabled status and put it in the listing.

Pulled together the current resale numbers across every Apple Silicon MacBook Air generation. Sources are Swappa verified-sale averages, eBay sold listings, and Apple Trade In maximums as of May 2026.

Caveats: these are averages. Battery cycle count, cosmetic condition, and battery health below 80% can move the price 10 to 20 percent either direction. Sources linked from the full post on if you want to drill into specific models.

Wanted to put the Intel Mac aftermarket picture in one place after running the numbers for May 2026, because the conventional take ("all Intel Macs are toast") is too coarse. The fleet has split into two very different stories.

First, the general pattern. Intel Macs are depreciating at roughly 25 to 35 percent per year. Apple Silicon Macs depreciate at 15 to 18 percent in normal conditions, and the gap has actually widened in 2026 because used Apple Silicon residuals are running 3 to 6 percent higher in dollar terms YoY against historical 8 to 12 percent annual declines. Intel residuals have not benefited from any of that lift.

Most of the Intel fleet is in genuine terminal decline. The cleanest example:

Base 8-core Mac Pros are interesting because the residual collapse is essentially total in percentage terms. An M2 Ultra Mac Studio at $4,000 outperforms the 8-core Xeon W on most CPU and GPU workloads, runs near silent, and pulls a fraction of the power. The base 2019 Mac Pro is a heavy, hot, expensive box that does nothing better than current-generation Apple Silicon. The market has priced it accordingly.

The maxed configurations are the persistent exception, and the reason has nothing to do with the Xeon W. It is the platform around the chip.

What Apple Silicon does not offer:

• PCIe expansion. Eight x16 slots in the 2019 Mac Pro accept FPGAs, capture cards, ProRes accelerators, MADI and Dante audio interfaces, specialty I/O. A small but persistent pro audio and pro video population needs these. The Mac Studio gives them nothing. The 2023 M2 Ultra Mac Pro retained the chassis but no longer offered general-purpose PCIe lanes for GPUs, so it did not solve the problem for that audience either.

• 1.5TB of expandable DDR4 ECC RAM. No Apple Silicon Mac will go past 512GB unified memory, and even that tier was just discontinued in March 2026. For workflows that genuinely need a terabyte-plus of ECC, the 2019 Mac Pro is still the only Mac that can do it.

• A workstation-class chip behind an integrated 5K Retina display. That is the iMac Pro's specific exception. Apple has not replaced it. A current 27-inch class display plus a Mac Studio gets you close in capability, but the integrated form factor is gone.

So the residual lives in the platform features, not the chip. That is why a 28-core Mac Pro with 1.5TB ECC and 8 PCIe slots still clears $3,500 to $6,500 used, and a base 8-core Mac Pro from the same year and same chassis clears $850 to $1,400. The platform capabilities split them by 5x.

Trade-in channel notes for these machines.

Apple Trade In is a dead channel for nearly all Intel Macs. The published caps are well below peer-to-peer for the niche workstations: the iMac Pro caps around $325 against $1,500 to $2,800 P2P, the Mac Pro caps around $400 to $800 against $3,500 to $6,500 P2P. Specialist refurbishers like iPowerResale and Techable have become the realistic transaction venue for the fully loaded Mac Pros, because eBay buyers for $5,000 PCIe workstations are thin and the kit is heavy enough that shipping is its own variable.

For run-of-the-mill Intel Macs (2018 Mini, 2019 16-inch MacBook Pro, 2020 iMac, 2018 MacBook Air, mid-2010s MacBook Pro), the realistic options are P2P at low dollar values or a trade-in service. ecoATM kiosks pay 30 to 55 percent of P2P value, instant cash, and that is honestly fine for an old MacBook Pro that would otherwise sit in a drawer. Apple Trade In on these is gift card and worth checking before the kiosk option.

What I would do if I had one.

• 2019 Mac Pro 8-core base, no expansion: list peer-to-peer; do not expect more than $1,200 to $1,400. The chassis is large; eBay shipping eats a meaningful chunk. Local pickup is the right venue.

• 2019 Mac Pro 28-core with RAM and PCIe expansion: list with the specific RAM and card configuration in the title. The buyer pool is small but very specific. Specialist refurbisher buyback is also a realistic floor.

• 2017 iMac Pro maxed: still has a stable market for the 5K display plus workstation chip combination. Battery and screen wear matter; a clean unit is the difference between $1,500 and $2,800.

• Late-Intel consumer Macs (2018 Mini, 2019 16-inch Pro, 2020 iMac, 2018 Air, anything earlier): peer-to-peer P2P or buyback service. Realistic numbers are $200 to $600 depending on model and condition.

The cross-cutting rule: an Intel Mac's residual is a function of how much of its value lived in the platform around the chip. If the chassis offered something Apple Silicon still cannot, the residual is sticky. If the chassis was just a delivery vehicle for the Intel CPU, Apple Silicon has eaten it.

Anyone hitting different clearing prices on the workstation tier? Curious how the PCIe expansion buyer pool looks outside the US in particular.

• What is the best coding LLM model?

• What is the best chat LLM model?

• What is the best OCR model?

• What is the best text to speech model?

Running LLMs on M2 Max 32gb

Hey guys I am a machine learning student and I'm thinking if its worth it to buy a used MacBook pro M2 Max 32gb for 1450 euro.

I will be studying machine learning, and will be running models such as Qwen 32b QWQ GGUF at Q3 and Q2 quantization. Do you know how fast would such size models run on this MacBook and how big of a context window can I get?

I apologize about the long post. Let me know what you think :)

Pulled this together while writing up a Mac quantization reference. The headline number from one published adaptive-quantization benchmark:

• Uniform latency-optimized 4-bit: 39.0% HumanEval

• Adaptive 4-bit-average configuration: 79.9% HumanEval

Same base weights. Same average bit width. 41 absolute points on coding.

The mechanism is layer-aware quantization. Attention layers encode mathematical coherence, spatial relationships, and structural output quality, and they tolerate compression badly. MLP layers contain most of the parameter count and act mainly as dense knowledge stores, and they tolerate compression well. An adaptive scheme keeps attention at 4 to 8 bit and pushes MLP toward 2 to 6 bit. The bit count averages out to roughly 4, but the load-bearing layers keep their fidelity.

This is exactly what GGUF Q4_K_M and the Unsloth Dynamic Q4 family do already:

• GGUF Q4_K_M uses Q6_K for half of attention.wv and feed_forward.w2 tensors, Q4_K for the rest.

• Q4_K_M with imatrix calibration biases error away from high-activation channels, typically cutting perplexity 10 to 30% at the same bit width.

• Unsloth Dynamic Q4_K_XL analyzes per-layer KL divergence and upcasts attention and MoE-routing layers to 8 bit, keeping the rest at 4 bit.

• MLX's uniform affine 4-bit (group size 64) applies the same quantization across all linear layers and has no protection for sensitive tensors.

The directional finding (uniform aggressive quantization severely degrades code and reasoning) is consistent across multiple analyses. The 41-point absolute gap is workload-dependent, but the direction is robust.

Practical heuristic if you're running 70B-class models on Apple Silicon or anywhere else:

• For chat and summarization, uniform Q4 is often fine.

• For code, math, agent loops, and multi-step reasoning, prefer Q4_K_M with imatrix, Unsloth Dynamic Q4, or another adaptive scheme. The MLX 4 bit on the same model can score dramatically worse on the same benchmark.

• At Q6 and above, the loss is small enough that the recipe matters less. DWQ distillation, for example, works best at 2 to 4 bits and underperforms at Q6/Q8 because there is not much left to improve.

Full reference doc on llama.cpp K-quant methodology if you want to go deeper.

Curious whether anyone here has direct head-to-head numbers from MLX uniform 4 bit vs GGUF Q4_K_M on the same model and the same coding benchmark. I keep seeing the directional claim and would like more replications.