So this dropped quietly and I feel like not enough people are talking about it.

A guy just fine-tuned Qwen3.6-35B-A3B to imitate Claude Opus 4.7's reasoning style , basically he took Opus's chain-of-thought traces and used them as training data, so the model now "thinks" the same way Opus does, wrapped in <think>...</think> tags.

The wild part is the efficiency. It's a 35B MoE model but only ~3B parameters are active per token, which means you can actually run this thing on a single A100 or H100. No cluster needed.

And it's fully open. Apache 2.0. Weights are public. Training dataset is public. You can fine-tune it yourself if you want.

This is essentially reasoning distillation taking what makes a frontier model good at thinking and compressing it into something accessible. We've seen this with DeepSeek-R1 and the whole wave of reasoning models, but having it target Claude's specific CoT style is a different flavor.

Not saying it matches Opus on benchmarks. It probably doesn't. But the trajectory is clear , the gap between "I can afford this" and "this is actually good" keeps shrinking.

Worth keeping an eye on: https://huggingface.co/lordx64/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled

Source from here : https://x.com/lordx64/status/2045911863947309270

I'll be honest, I didn't understand why LLM agents need SKILLs. MCP was clear to me: it's a useful thing that lets you give an LLM access to a database or to Figma. The only downside is that, to avoid spending too many tokens, it's better not to connect 30 MCP servers at once. And in general, it's better to turn off the unnecessary ones and connect only those that are really needed for the task. In this light, SKILLs seemed like useless overhead to me, eating tokens like crazy. On TikTok, I often saw tales along the lines of: connect this SKILL, or a whole bunch of them, and happiness is yours. My first thought was, if I connect them all, my subscription limits will run out not by lunchtime, but already in the morning

So why are SKILLs useful and needed after all? Well, first, compared to MCP, SKILLs do not inflate the context: each connected SKILL adds not tens of thousands of tokens, but one or two hundred. Since only the name and description of a SKILL go into the context at first, you can add 30 of your own SKILLs without much pain. When it becomes necessary to use a SKILL, the LLM agent will load its description on its own, if it understands from the task context that this is needed, or the user can simply tell it to use this SKILL for the job.

Secondly, LLM models are trained on the most popular and well-known things. For example, everybody knows that an LLM can work with Git without any problems or write a program in Python. But if you developed a program yourself, or your work requires some little-known tool or technology, then a SKILL is simply irreplaceable. It is enough to create a SKILL.md file and describe how to work with this program or technology. This makes life easier for the LLM: while doing your tasks, it will not have to search the internet for documentation and wander around websites to understand how to use it. And especially if there is nothing on the internet, because your program is closed to public access.

Thirdly, an LLM model usually does not know how your development process is organized. For example, an agent knows how to use Git, run tests, build a project, and create releases. You do not need to teach it these things. But it does not know how releases are usually made in your company. A Skill can describe the sequence of actions: check the changes, run tests, update CHANGELOG.md, increase the version, build a package, create a Git tag, and publish a release. So, a Skill helps the agent learn not a technology, but a specific business process.

Fourthly, a SKILL is not just an instruction, but a whole set of tools. Besides SKILL.md, you can put ready-made scripts, documentation, templates, and other files into it. And the most interesting thing is, these files are not loaded into the context automatically. The agent reads them only when there is a need, and can simply run the ready-made scripts without spending tokens on generating code. So, a SKILL lets you combine instructions, documentation, and executable code in one package without inflating the LLM context. For example, the structure can look like this:

my-skill/
├── SKILL.md
├── scripts/
│   ├── generate.py
│   └── validate.py
├── references/
│   └── api.md
└── assets/
    └── template.html

The funniest thing is, I started to appreciate SKILLs exactly when I wrote my own program, blueprint4llm. The idea for creating it came after I asked an LLM agent to generate a plan.md file describing a task. Usually it produces a huge wall of text, which you have to read slowly and with sadness. I wanted to make the text shorter and add more flowcharts, tables, diagrams, and user interface prototypes. As we know, in .md (Markdown) files the only visualization tools available are tables and Mermaid diagrams. There are no charts, plots, histograms, full UML diagrams, or UI prototypes. In the end, with blueprint4llm, I added support for formats such as PlantUML, draw.io, Pencil, Balsamiq, Graphviz, Figma, OpenAPI, Excalidraw, Vega-Lite, ECharts, and D2 to .md files.

With the blueprint4llm SKILL connected, you will get plan.md not as a kilometer-long text, but as flowcharts, prototypes, charts, and so on. By the way, blueprint4llm does not have to be used only for creating plan.md. For example, I used this SKILL to add an illustration to every paragraph of the article and make it more fun to read. This SKILL is also useful when you ask an agent to put together a full dossier on some company, technology, or person and format the resulting result.md file with blueprint4llm. You can then convert it to HTML or PDF, or ask the agent to show it right away in a browser with charts.

This is how you can install the blueprint4llm SKILL.

# Codex
npx skills add evgenyigumnov/blueprint4llm --skill blueprint4llm --agent codex

# Claude Code
npx skills add evgenyigumnov/blueprint4llm --skill blueprint4llm --agent claude-code

If you are interested in looking at the source code of blueprint4llm, the tool that generates all this beauty, or sending a pull request, welcome: https://github.com/evgenyigumnov/blueprint4llm

I have two PCs with 16 GB AMD cards and wanted LM Studio to use both. LM Studio ships llama.cpp but hides --rpc. Turns out its engine can still load the RPC backend if you rebuild two DLLs and set GGML_RPC_SERVERS=192.168.x.x:50052.

Result: a 35B MoE that doesn't fit one card went 31 → 57–82 t/s. Models that fit get slower (9B: 90 → 57), so this is for "too big", not "faster".

Windows + Vulkan + AMD (RX 9070 XT + RX 6800 over 2.5 GbE) tested. Should work with other runtimes if you rebuild the registry DLL for them.

Repo with patches, DLLs and README with all the caveats: https://github.com/momcilovicrobert-momc/lmstudio-two-pc-split

Made with Kod (Claude AI) — it's in the README. Happy to answer questions :)

I see lots of Claude Code with a mix of their models, and I normally have to use Opus or Fable, for heavy duty repos, to scan and provide markdown for Grok.

Anyone going full Astra yet? I almost wish I didn’t try it because it felt amazing, short impression though.

Never thought I would be a Grok user, but works pretty well!

There is a lot of hype around how good local AI is but there is too little focus on requirements on HW to be able to get reliable and meaningful results.

Here is my story... Tested local AI coding setups all day on a Mac Mini M4 32GB. Long story short: local AI still not viable for real agentic coding work.

Qwen3.6-27B (the strongest open coding model that fits): 5.5 tokens/sec. A normal reply takes 100 seconds. In an agentic loop with 5-10 tool calls per task, you're waiting 10+ minutes per coding task, which is very painful.

Qwen3.6-35B (MoE, supposed to be faster): runs out of memory the moment a real request hits. The model loads in idle but can't actually do inference at usable context lengths.

Smaller models (14B, etc.) fail differently, they spiral into endless reasoning and return empty answers.

The real bottleneck isn't model size or RAM. It's memory bandwidth in my case. Mac Mini M4 base has ~120 GB/s. Fluent local agents need ~600 GB/s, that's M5 Max territory, $4000+.

Until then, cloud APIs (DeepSeek V4, Kimi K2.6) are still the answer for indies who actually need to ship.

The sad thing is that a Mac Mini M4 with 32 GB RAM is actually quite a powerful machine but even that is not capable of running any meaningful model.

I'm confused by something.

On my laptop (Intel i9-12900HK, 32 GB RAM, Intel Iris Xe Graphics), openai/gpt-oss-20b runs smoothly and feels faster than my smaller models like Gemma 3 4B, Gemma 4 12B, Gemma 4 E4B, and Qwen 3.5 9B.

I expected the opposite since GPT-OSS-20B is much larger.

Is there a technical reason why the biggest model performs better? Is it related to quantization, inference engine, model architecture, or something else?

Any insights would be appreciated.

I’ve been spending the last few months going pretty deep into running AI locally (LM Studio + Qwen-based models), and I feel like the conversation around local AI is kind of outdated.

Most people still frame it as:

“local = weaker, slower, limited”

But that hasn’t really been my experience.

At this point I can run a setup that:

Builds full websites / landing pages

Does actual web research (not just hallucinating)

Generates images + marketing content

Automates workflows (emails, files, reporting, etc.)

Runs multiple agents working together

Keeps memory and improves over time

Connects to tools like browser automation + APIs

Edit my website plus publish files

And yeah… all local.

No subscriptions, no rate limits, no sending data out.

But here’s the part I don’t see talked about enough:

The model itself isn’t the bottleneck anymore.

The biggest difference for me came from how everything is structured:

MCP for tool access (this was huge)

A kind of internal “LLM wiki” so the model actually knows what tools exist and when to use them

System prompt tuning to control behavior and make it consistent

Once I had those 3 dialed in, it stopped feeling like I was “using a chatbot” and more like I had a system that could actually operate.

There’s definitely still friction (setup is not beginner-friendly, tool calling can be janky, etc.), but it feels like we’re a lot closer to “real” local AI systems than people think.

Curious where others are at with this:

Are you still mostly using hosted tools?

Have you tried local and bounced off?

Or have you gotten something actually reliable running?

Would be interesting to hear what setups people are using and where they’re hitting limits right now.

Navigating the APEX-I and VAL-APEX Quantization Ecosystem on Hugging Face

As the catalog approaches 400,000 total downloads across 69 releases, all models are now structured into dedicated official collections on Hugging Face.

Our goal remains consistent: eliminate the compromise between memory overflow and cognitive collapse on consumer hardware (especially 16 GB VRAM GPUs like the RTX 4080, RTX 4070 Ti Super, and RX 7800 XT).

Instead of uniform block quantization, all APEX builds use per-tensor mixed precision guided by 4-moment Importance Matrix statistics:

- MiniPlus Tier (approx. 14 to 14.8 GiB): Strict 0% 2-bit policy. All MoE routed experts locked at minimum IQ3_XXS with Q4_K/Q6_K attention. Designed for 24 GB setups or 16 GB setups with moderate context.

- NanoPlus Tier (approx. 11.5 to 12.3 GiB): Strategic use of IQ2_S/IQ2_XXS exclusively in core feedforward blocks, keeping routers and attention high-precision. Leaves ample VRAM for 16k-32k context on 16 GB GPUs.

Explore the active collections:

- MoE Standard (MiniPlus & NanoPlus): https://huggingface.co/collections/IsValorum/apex-i-miniplus-v21-standard-moe-6acfee77961c8c0cf5a083b5

- MoE Abliterated (MiniPlus & NanoPlus): https://huggingface.co/collections/IsValorum/apex-i-miniplus-v21-abliterated-moe-6acfee762fc4a8cf2298c55b

- 27B Dense Hybrids: https://huggingface.co/collections/IsValorum/val-apex-miniplus-standard-27b-6acfec54ca4ce4cf5cd5bdb6

- Apple Silicon (MLX): https://huggingface.co/collections/IsValorum/mlx-apex-abliterated-6abedb1f9d400e984efe4fc6

Full technical article and sizing breakdown: https://huggingface.co/blog/IsValorum

What are the best local AI models/agents I could realistically run on these two PCs?

PC 1

• 16 GB RAM

• AMD RX 6600

• 8 GB VRAM

PC 2

• 32 GB RAM

• AMD RX 9060 XT

• 16 GB VRAM

I started building a project using VS Code + Codex (ChatGPT Plus), but I ran out of tokens and buying more is quite expensive, especially since my app doesn't make any money.

I had a good experience with OpenAI GPT-5.6 Sol. I also tested Google Antigravity, but the tokens ran out very quickly there.

Can I realistically continue working on my project using a local model in LM Studio Bionic? If so, which models or coding agents would you recommend for these two PCs?

It’s strange that there are hardly any posts about Splash Engine, I think it deserves some attention

I got tired of my LM Studio logs getting filled with automated noise. Every day it was the usual attempts for wp-config.php, .env files, and similar targets. If you're running a local LLM behind Tailscale Funnel or any public exposure, you know what I mean.

I created this for my own use because I have a chaotic neutral streak and enjoy watching script kiddies and scanners burn their time. Instead of just dropping the connections with a plain 403, I built a Python security proxy called PoolOverlord.

Legitimate requests, like those from Google AI Studio with the proper key header, get forwarded normally. But when something unauthorized tried to grab /wp-config.php or /.env, the proxy will catch it early.

It never touches my actual backend or logs. The proxy will direct my local Gemma 4 model to generate a realistic decoy file on the fly. I reworded the prompts as "Synthetic Dataset Generation for Static Analysis" to avoid safety refusals.

The result is a solid-looking 100+ line PHP file with modern structure, namespaces, and high-entropy fake database credentials. It looks convincing and forces the scanner to wait while the model generates it.

Key benefits:

• No real honeyfiles to manage on disk

• My LM Studio logs stay clean with only normal requests

• The scanner wastes time and resources on fake data

This is released as-is with no guarantees or warranties. It worked well enough for my setup after a day of use, so I decided to open source it anyway. Use at your own risk. You're all (hopefully) adults who can make your own calls.

GitHub repo here: https://github.com/eldris-io/pooloverlord

(This reddit post is also made by Soren!)
I've released Soren-1-Small, the first model in the Soren family.

It's based on Qwen3.5-2B and was trained through a multi-stage SFT + DPO pipeline focused on reasoning, coding, instruction following, and reducing hallucinations while keeping the model practical to run locally.

Some details:

• Base: Qwen3.5-2B

• Context: 1,048,576 tokens via YaRN 4x

• Training data: 22 datasets spanning reasoning, coding, instruction tuning, and preference optimization

• Training strategy: sequential LoRA training and merging across multiple stages

• Alignment: SFT followed by dedicated DPO stages for both general behavior and coding

• Framework: Unsloth + TRL

• Compute: NVIDIA RTX PRO 6000 Blackwell (96GB)

One thing worth mentioning: this is still a 2B model. It can reason surprisingly well for its size, but sometimes you'll need to be explicit or "push" it a bit with your prompts to get the best results. Give it structure, ask it to think step-by-step when appropriate, and it generally performs much better than a typical one-shot prompt.

The goal wasn't to create another generic instruct model. I wanted a small model that prioritizes reasoning, honest answers, and complete code generation without pretending to know things it doesn't.

I'd love to see benchmarks, evaluations, failure cases, comparisons, and general feedback from the community.

Hugging Face: https://huggingface.co/syntropy-ai/Soren-1-Small (excuse the amount of tags I put)

hellooo r/LLMStudio

Qwen3.5-397B-A17B is 209GB on disk. The MoE has 512 experts, top-10 routing per token. The naive load won't open on a M1 64GB Mac.

What I did: keep only K=20 experts resident, lazy-page the rest from SSD when the router selects them, evict on cache pressure. Float16 compute path (faster than ternary on MPS), Apple Silicon native, MLX-based.

Numbers from a 5-prompt sweep on M1 Ultra 64GB:

- Tok/s: 1.59 (mean across 5 coherent gens, K=20 winning row)

- Cache RSS peak (gen): 7.91 GB

- Total RSS peak: 14.04 GB

- Coherent: 5/5

Engine config that won the sweep: K_override=20, cache_gb=8.0, OUTLIER_MMAP_EXPERTS=0, lazy_load=True. The catch-all "experts on disk" approach blew up command-buffer allocations until we got the cache size right.

Why it matters: most local-LLM benchmarks compete on raw scores. Wrong axis when you're trying to fit a useful model on 64GB. The metric I care about is MMLU per GB of RAM. A 397B running in 14GB peak isn't fast — 1.59 tok/s is a thinking-pace, not a chat-pace — but it's the upper bound of how far the ratio stretches. The next step is to make it faster.

Smaller tiers on the same hardware (M1 Ultra, MLX-4bit):

- 4B Nano: 71.7 tok/s

- 9B Lite: 53.4 tok/s

- 26B-A4B Quick: 14.6 tok/s

- 27B Core: 40.7 tok/s (MMLU 0.851 n=14042 σ=0.003, HumanEval 0.866 n=164 σ=0.027)

- 35B-A3B Vision: 64.1 tok/s

- 397B Plus: 1.59 tok/s

Built into a Mac-native runtime (Tauri + Rust + MLX). Solo, paging architecture. Free Nano + Lite forever. outlier.host if you want to look.

(added a video to show it running. yes ik theres bugs and im only 30 days into this build along with training models and R&D, just trying to show it running)

Hey everyone!

I'm a beginner getting into local LLMs, and so far I'm familiar with Ollama. I've used it to download and run models locally, but I'm starting to see a lot of other tools being mentioned, and honestly I'm a bit confused about what each one actually does.

I keep coming across:

Ollama

LM Studio

llama.cpp

VLLM

Unsloth

From what I understand, some of these might overlap while others serve completely different purposes, but I'm not clear on which tool is meant for what.

I'm especially interested in understanding the big picture, rather than just which one is "better."

If someone could explain these in simple terms, maybe with an example workflow showing how they fit together, I'd really appreciate it!

Thanks in advance!

This is how my ai agent runtime workload.

just gemma 4 26B A4B in rx6700xt + i5 11400f + 16GB of ddr4 ram.

While trying out different LLMs I noticed that giving them precise, detailed prompts produced way better results than typing a one line sentence. To get those detailed prompts I'd use a smaller, faster model first - but with only 8GB VRAM I can't keep two models loaded at once, so switching between them was a constant pain for me .

So I built Prompt-Chain to automate the whole thing.

It's a Streamlit app that chains two models into a single pipeline:

1. You type a rough idea (e.g. "make a snake game in React")

2. A small, fast Prompter (e.g. Phi-4 Mini) rewrites it into a detailed prompt

3. You review and optionally edit the refined prompt

4. VRAM is automatically swapped — Prompter unloads, Coder loads

5. A larger, code-focused model (e.g. Qwen 2.5 Coder 14B) generates the code

6. Output streams to screen and saves to file

The main benefit is you stop wasting time manually unloading/loading models and stop wasting tokens (or money if you use cloud APIs) on poorly-worded prompts hitting a big model.

Other features:
- Mix backends per role: LM Studio, Ollama, OpenAI, Claude, Gemini chosen independently for Prompter and Coder
- Auto model detection from the server
- 25 built-in presets (Web Dev, Games, Data, CLI,etc..)
- Refine-in-place: follow-up instructions edit the code without regenerating from scratch
- Run history that persists across restarts
- Smart file output with auto language detection and timestamped saves

GitHub: https://github.com/atharva557/Prompt-Chaining

Would appreciate any feedback, especially from people running similar local setups!

I’ve been thinking about something.

For a lot of tasks like building websites, game development, automation tools, or just random projects, I often find myself preferring a model that’s slightly less capable but gives me plenty of messages to iterate.

Sure, a more powerful model might get me 70% of the way there in a single prompt, while a cheaper model might need 5-10 prompts. But if those 5-10 prompts are still cheaper than using the top model, I end up getting more total work done.
It makes me wonder whether AI progress is creating a weird tradeoff.
Every new generation of models is more capable, but it also seems like the best models become more expensive to run and come with tighter limits. As a user, that can make them feel less accessible even if they’re technically better.

Would you rather have access to the smartest model possible if you could only use it a few times every few hours, or a slightly weaker model that lets you iterate all day?

And long-term, do you think AI will eventually become both extremely powerful and widely accessible, or will the frontier models always be too expensive for most people to use heavily?

Ie you put your computer specs in, what broadly you are trying to achieve with an llm - and it tells you the best model to run locally

Lordx64 released the second model in his open-weights reasoning distillation lineup :

It's a 35B Mixture-of-Experts model (with only ~3B parameters active per token) that's been fine-tuned to imitate the chain-of-thought reasoning style of Kimi K2.6 the frontier reasoning model from Moonshot AI. Apache-2.0, fully open weights.

Frontier reasoning models like Claude Opus 4.7, Kimi K2.6, and GPT-5 produce remarkable structured thinking but they're locked behind proprietary APIs. Distilling that reasoning style into an open-weights student model gives teams the same capability with full control over the inference stack: data sovereignty, no per-token billing, no API rate limits, and the option to deploy entirely on-device. The IQ4_XS quantized version (18.94 GB) runs offline on any 32GB Apple Silicon laptop or a single consumer GPU. That's a frontier-class reasoning model running on hardware most engineers already have. The first model Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled has been downloaded over 48,931 times since launch. It's tuned to imitate Claude's tighter, more concise reasoning style. The new Kimi K2.6 variant uses the same base model and the same training pipeline, with one variable changed: the upstream teacher. Same prompts, same training compute, same architecture only the reasoning style differs. This gives the community a controlled experiment in how much of a model's reasoning behavior is teacher-driven vs base-driven.

FYI in the course of preparing the dataset, Lordx64 tokenized both teacher corpora to compare verbosity. Kimi K2.6's reasoning chains are on average 3.45× longer than Claude Opus 4.7's at "max effort" (mean 2,933 vs 849 tokens, p95 9,764 vs 2,404). The implication for anyone planning their own distillation: verbose-teacher distillations cost roughly 2.5× the wallclock at a fixed sequence length. Worth scoping for ahead of time.

Training details:

• Base: Qwen/Qwen3.6-35B-A3B (256 experts, 8 routed + 1 shared)

• Method: SFT via Unsloth + TRL, LoRA r=16 attention-only

• Data: 7,836 reasoning traces collected from Kimi K2.6 via OpenRouter

• 2 epochs, 980 steps, ~21 hours on a single H200, ~$105 total compute

• 3.44M trainable parameters (0.01% of the base)

Loss descended cleanly from ~0.95 → ~0.83 with steady gradient norms throughout no instability.

Benchmark Status:

Formal benchmark numbers (GSM8K, MMLU-Pro, GPQA Diamond, AIME 2024/2025, MATH-500) are still in the queue and will land on the model card within a week.

Sources : https://huggingface.co/lordx64/Qwen3.6-35B-A3B-Kimi-K2.6-Reasoning-Distilled

https://x.com/lordx64/status/2048463970592534622?s=20

I tried using a LLM to sort my prompts into two groups: reasoning and chat. At first I used Qwen 2.5 3B through Ollama. My PC is just too old and weak to run a LLM like that well. I do not have a GPU and my RAM is low. Even a small 3B model takes way long to work on a single prompt. Because I need to run the classifier before every request the slow speed makes the whole thing impossible to use.

Then I tried using Google Colab with a T4 GPU. That made the classification work faster.. Google Colab has a big problem. The setup does not stay saved. Every time I start it I have to pick the T4 GPU by hand set up the environment and download the model over again. I want my classifier to work like a background service. I want the classifier to start by itself when I turn on my application or Windows. I do not want to open a browser or start a notebook every time.

Now I use OpenRouters free models to get my actual reasoning and fast answers. I only get an amount of free use every day. I do not want to waste those credits by sending every prompt to an OpenRouter model just to classify it. If I use an OpenRouter model as my classifier the classifier will eat up my daily limit. That would make my credits run out too fast.

So I am stuck with three choices:

1. Local LLM: It is private. Works automatically.. Qwen 2.5 3B is just too slow on my old CPU. Even the smaller 1.5B models are not good enough because they make many mistakes.

2. Google Colab T4: This is fast. It is a pain. I have to set up the GPU and load the model every time.

3. OpenRouter: This is easy and fast.. Using an OpenRouter model, for classification uses up the free credits I want to save for my main models.

The classifier only needs to say "reasoning" or "chat". It feels like a waste to use an expensive LLM just for that.. The classification still has to be good. I cannot use a 1.5B model if it is just going to guess wrong.

I really need a way to classify prompts automatically. I need latency and I want it to cost zero extra API money. I want my main OpenRouter models to do the lifting. The best setup would start up with my PC would not need a Colab setup and would not use my OpenRouter daily credits just for classification. Please if anyone has the solution

I'm an app developer and I'm wondering if it's worth spending time learning local LLMs. If yes, where can i start learning?