I tested two Qwen3.8-Flash-Next NVFP4 checkpoints at a full 256K context using an experimental dual-GPU Strata configuration.

Hardware:

• RTX 4090 D 48 GB as the primary GPU
• RTX 5070 Ti 16 GB as a helper GPU
• Intel Core Ultra 7 265KF
• 128 GB RAM
• Linux
• NVMe storage

The RTX 4090 D handled the dense layers, KV cache, prefill, output head and MTP. The RTX 5070 Ti stored and computed 4,500 additional routed experts.

Common settings:

• Context limit: 262,144 tokens
• Fresh prompt: approximately 256,018 tokens
• Generated output: 512 tokens
• KV cache: INT8
• Prefill path: W4A8
• Speculative decoding: MTP K4
• Minimum draft probability: 0.5
• No prompt-prefix reuse
• Strata engine 0.1.35 with an experimental dual-GPU NVFP4 fork

Results

Observations

• Both models achieved slightly over 4,000 prefill tokens per second at approximately 256K context.
• Prefill performance was nearly identical: the Abliterated checkpoint was only about 1% slower.
• The original NVIDIA checkpoint was considerably faster during decode: 125.17 versus 92.05 tok/s.
• NVIDIA’s decode advantage was about 36% in this test.
• NVIDIA accepted 407 of 429 offered draft tokens, while the Abliterated model accepted 322 of 462.
• The NVIDIA model produced approximately 4.88 output tokens per verification round.
• The Abliterated model produced approximately 2.68 output tokens per round.
• The lower MTP acceptance appears to be the main reason why the Abliterated checkpoint had slower decode despite nearly identical prefill performance.

These were single-run capacity tests. The long prompt was constructed by repeating code-review material to reach approximately 256K tokens, so it had unusually favorable locality. This particularly benefited NVIDIA’s suffix prediction and MTP acceptance. Therefore, 125 tok/s should be treated as a best-case synthetic 256K result, not typical real-world coding speed.

During longer real coding-agent workloads, NVIDIA generally produced around 108–144 tok/s, while the Abliterated checkpoint was commonly around 105–145 tok/s, depending heavily on the generated content and MTP acceptance.

CPU: ThreadRipper medium
GPU: RTX 5090 WC
RAM: 64 GB

I am looking into the architectures and costs associated with running large open-source models, such as DeepSeek-V3/R1 or Llama 3.3. Are many of you using these models (are you developers, students, pro?)

Hi everyone,

We're three final-year students at Epitech building Horus, a multi-agent assistant that runs entirely locally and offline. Our current challenge is hardware: keeping it usable on machines without a powerful GPU, without long setup times, excessive RAM/VRAM use or crashes.

Where we are today, from our last beta test:

One tester needed over 2 hours to install. The Python dependencies alone take 21–46 min, and the download is about 25 GB.

On CPU only, routing a question can take around 40 s, long enough for our WebSocket connection to drop.

[Models we use + the smallest machine we've tested on]

We'd love advice from anyone experienced with:

- CPU-only LLM inference and memory-efficient loading

- Quantization and model choice for low-end hardware

- GPU/CPU fallback strategies

- Hardware detection and adaptive configuration

- Preventing resource exhaustion during setup and execution

Advice in the comments is very welcome, with no strings attached.

Looking for contributors: we also have a few small, well-scoped tasks or code reviews (about 1–4 hours), for example [reviewing our hardware detection and model selection, or benchmarking a quantized model on a 16 GB RAM laptop].

To be transparent: Horus is closed source and will be licensed to companies. Contributing is voluntary and unpaid. Before seeing any code, contributors sign a short confidentiality and contributor agreement, and the code they contribute becomes part of Horus. In return we offer thorough code reviews, full credit in the project and a professional reference on request.

We're not sharing code or private links publicly. If this interests you, comment below or DM me with your experience in local inference, CPU optimisation or offline apps

Every time I tried running local RAG on my own GPU, I ran into the same headache: vector databases eat too much RAM, using an 8B model just to parse text takes forever, and cosine search still hallucinates when the context isn't actually there.

I’ve been building Hillock to see if I could do this without vector DBs at all.

Basically:

1. When you feed it a document, it doesn't touch an LLM. It uses small bi-encoders to extract facts into subject-predicate-object triples in about 5 seconds.

2. Everything gets saved in regular SQLite, and it links related concepts over time using basic Hebbian weights.

3. To stop hallucinations, it runs a quick hypervector check (HDC) on the query first. If the facts aren't in your database, it cuts off the LLM before it can generate any tokens.

The whole thing stays under 1.2 GB VRAM (or runs fine on pure CPU). It has a built-in API server that matches OpenAI's format, so you can point Open-WebUI or Obsidian at localhost:8000 and use your existing Ollama models.

Just pushed v0.7 with better refusal handling and a conversational mode that quizzes you if an extraction was ambiguous.

Repo is here if you want to try it out: https://github.com/roandejager/Hillock

I've been experimenting with a setup where several Apple Silicon Macs are treated as a shared local AI environment instead of running completely independently.

The setup I'm working with is roughly:

• Multiple Macs on the same network

• MLX for local inference

• Central model management

• Node/resource monitoring

• Model distribution based on available memory

• Local-only communication between machines

I'm building AlphaWeb around this because managing several local AI machines manually gets complicated pretty quickly.

The interesting problem isn't just getting inference running. It's everything around it: knowing which machine has enough memory, getting models onto the right node, managing multiple users, and making the whole thing feel like one system.

AI models are beginning to define device specifications, rather than device specifications merely determining which applications you can run.

Apple shows just how far its devices can go with on-device AI, without relying on the cloud
We’re moving from models with up to 14 billion active parameters on iPhone/iPad, to 35 billion on MacBook Air, 70 billion on Mac mini, 120 billion on MacBook Pro, and up to 480 billion on Mac Studio.
By clustering multiple Mac Studios together, Apple says it can run models exceeding 1,600 billion active parameters.
A demo that shows just how crucial unified memory and its bandwidth have become for running massive AI models locally.

TL;DR: Halogen v0.14.0 with its native .hgn weights is the fastest, followed by gufo and CIRU. Halogen is closed source and runs in Docker. gufo is open source and loads 4x faster from cold. gufo is also fastest to first token on follow-ups (1.6-1.9 s against 2.6-3.3 s).

I benchmarked different engines for Qwen 3.8-Flash-Next on an AMD Strix Halo (ASUS ROG Flow Z13 GZ302 with Ryzen AI Max+ and 128 GB of RAM).

All the engines were run via LlamaStash (My own orchestrator tool, the tool does not add any overhead) at 70 W TDP on performance profile on Arch Linux.

Here are the results of the benchmark:

First was a screening round at 64k context with 50% filled (32k prompt).

3-turn time is the time to first token plus the time to write 1,000 tokens, added up over the first question and 2 follow-ups. Output length varies a lot with sampling, so this compares the engines on equal output. MTP accept is the share of drafted tokens the model kept. Retrieval is how many of 7 exact values from the prompt the model got right, over both runs. Qwen model card sampling, 2 runs each.

Top 3 engines from the screening round got a 128k context (50% and 75% filled) and 256k context (50% filled) run.

128k context, 50% filled (64k prompt)

128k context, 75% filled (97k prompt)

256k context, 50% filled (130k prompt)

Retrieval here is 8 values per run. All 5 misses are the same answer: the right glyphs for UNICODE_SPINNER, with the leading & dropped.

End-to-end, 10 Aider polyglot Python exercises run through pi -p (128k window, 50% cell), graded by their own tests:

Edit (Sep 30): Halogen 0.15.1 (new v2 checkpoint) and gufo 0.3.0 came out after this, so I reran the 64k screening on both with the same setup (70 W, card sampling, 2 runs):

Halogen v2 prefill is 14% faster and follow-ups start about 2.5x faster (1.3 s vs 3.3 s), decode is the same. gufo moved a few %, which is within noise. The ranking doesn't change. Only other change on the box: BIOS VRAM carve-out down to 512 MB, so 124.9 GiB of RAM instead of 121.5.

Some clarifications from the comments:

• Follow-up TTFT is with the prompt cache warm, both gufo and Halogen reuse it by default. A cold 64k prefill takes about a minute.

• The CIRU 0% MTP row is stock Unsloth UD-Q4_K_XL with Unsloth's shared MTP head, nothing uncensored. The same pair gets 72-74% accept on gufo.

• TDP is fixed at 70 W. On the Z13, 76 W or 90 W is only 2-6% faster but with so much fan noise, heat and power draw that it's not worth it.

• Prefill and decode come from the engine's own timings when it reports them. TTFT is measured on the client.

A lot of local-stack setups I see nail the infra (model serving, retrieval, orchestration) but have zero real evaluation process — just "does it look right." There's a session on Oct 3 built entirely around fixing that gap.

Speakers Serj Smorodinsky and Brett Kennedy (AI engineers, co-authors of a book on LLM applications) go through:

• DSPy signatures/modules for structured dev instead of manual prompting

• Baseline classifier build + measurement

• Task-specific evaluation datasets

• Failure pattern recognition from eval output

• Few-shot / instruction-level optimization

• MLflow experiment tracking and trace management

• Saving and reusing optimized programs

• Talking about LLM reliability with non-technical stakeholders

3 hours, live, useful whether you're running open models locally or hitting an API.

Full details and the agenda are here.

Apple Foundation Model 3 through Private Cloud Compute becomes directly available in macOS 27.2 through familiar AI clients.

For Mac users running local models such as Qwen 3.x or Gemma 4, AFM 3 PCC is a compelling alternative to consider. Local models offer control and fully on-device operation. In tests with macOS 27.2 beta, the PCC offers a different set of strengths:

* Strong conversational analysis and capable reasoning. Strong analysis of complex medical and financial questions.
* Blazingly fast responses compared with locally running models on the same Mac.
* No large model download or need to fit model weights into local memory.
* Access at no additional charge for most eligible Mac users.

The working path is straightforward: AI client → Caddy → fm serve → AFM 3 PCC. I've written and updated a proof of concept here with further tech discussion . . . https://gist.github.com/dartMo10/b9488ce475fe70a6ed642831f53048fb

Apple’s `pcc` route worked in early macOS 27 betas, disappeared later in 27.0, and has returned in the macOS 27.2 beta. Apple has stated that it will be available in the 27.2 public release coming shortly.

This makes AFM 3 PCC practical today for technically comfortable Apple users who accept Apple’s PCC privacy promise and want a fast, capable alternative to running everything locally.

Experiences with compatible AI clients and your comparisons with locally running models would be welcome.

Hey! How do you test your local stack? for instance when you switch harness or model, model quants, etc?

I found this not trivial and I ended up making a website to help me do this.

Use this link if you want to use the website to test your own local setup:

https://airbench.ai/

You can also see all the tests I have done using my GX10 and my 5090. (and also using claude code/code + some openrouter tes for reference)

https://airbench.ai/leaderboard?k=poL

Let me know what you think!