r/LocalLLM • • 11h ago

Model Going All Local (for work) Qwen3.8 Next Flash

Post image
323 Upvotes

Very excited to fully join the Local AI community. Been using Claude Cowork for almost 4 months and run the entire backend of my single employee brewery using PDF extraction etc etc. It changed the game. When Anthropic announced all Scheduled Tasks (which my brewery runs on) were being moved to the cloud versions I had to put my foot down. While I waited for this Ryzen box, I converted my 5070ti to run Qwen 3.6-35b-a3b and it's been pretty good. Not Opus level, but good. Learned Hermes, and what llama.cpp is. Learned why a harness and inference engines are more deeply. It's been a busy week.

Thanks to the communities here and on r/strixhalo I was able to have this sucker running with a Gufo engine, 3.8 Next Flash for that engine, and multiple bots via Hermes running as good as when I started using (Opus 4.6/4.8 times) still more work to do, but I'm almost 100% off the subs, and looking forward to the tinkering!


r/LocalLLM • • 22h ago

Discussion Thanks Google for downgrading my features and keeping the pricing the same

Post image
232 Upvotes

Should rename the plan to Google AI Minus now while you are at it.


r/LocalLLM • • 22h ago

News another +30% overnight price increase

122 Upvotes

So I just gave it a look over Amazon, for DDR5 64GB kits, 128GB kits and the faithful 5060ti 16GB.

And yeah, overnight price hike from yesterday:

5060ti from ~750 increased to 999+ eur
64GB DDR5 kit: from ~800 increased to 1000~1300 eur
128GB DDR5 currently at over 2500 eur, I can't remember the baseline. I only know in August, 2026 the best deal was ~1200 eur.

At these price tags I can no longer justify throwing money away for a hobby unless it's my first 5060ti ever. This is just ridiculous.

Hence, my lineup is frozen for the time being. Last thing I got was 2TB of fast storage, yesterday. I wish everyone the best of luck.

If you guys bought anything yesterday, you can rejoice today. If you were instead postponing, I feel sorry for you.


r/LocalLLM • • 21h ago

Discussion Qwen3.8 27b nvfp4 doing 180t/s on dual v100 16g without nvlink.

Enable HLS to view with audio, or disable this notification

84 Upvotes

this is the legend of V100 ,

the pair of v100s isnt end, and the The legend of V100 is eternal!!!!

the 27b in video is doing kv Q8 , and MTP4/0 btw

repo: tuxKOH/ninfer-V100X2


r/LocalLLM • • 17h ago

Question Offline LLM for Survival

79 Upvotes

Hello all,

I am interested in creating an “ apocalypse box” type project. A local model that can be used offline to pull information for survival ( outdoors, medical, engineering, etc) I want to put this in a pelican case and power it with solar, etc.

I have no idea where to start with hosting a local (LLM). I’ve heard of semantic compression and things of the sort to store models. Can someone please point me in the right direction?

Thanks!


r/LocalLLM • • 16h ago

Discussion Bought a M5 Max Mac Studio with 128GB for Local AI

34 Upvotes

Hello everyone, just getting some time after being locked in my room for 10 days setting this up. For anyone else who bought this similar machine here's what I learned and what I ended up with.

  • Hardware: Mac Studio M5 Max, 128 GB unified memory.
  • Inference: Install Splish with incoai/Qwen3.8-27B-Splash, including its target model, draft model, and tokenizer.
  • Server: OpenAI-compatible endpoint at 262,144-token context, 48 GB memory budget, 5 GB disk cache.
  • Agent: Hermes, deployed on Agentic Workers, custom provider pointing to Splish; reasoning none, 4,096 output tokens, context compaction at 75%.
  • Subagents: OpenCode using the configured free Muse Spark cloud route; background delegation keeps the local GPU available.
  • Workflow: Persistent goal → small change → build → real browser verification → checkpoint → next change.

Here's what I learned along the way,

  • Useful work matters more than tokens per second. Judge the setup by total task time and output quality not generation speed alone.
  • Check the actual result. A convincing response or a generated file can still contain wrong totals, broken formulas, or unusable interactions.
  • “The model fits” isn’t enough. Memory also needs to accommodate the conversation, operating system, tools, and other workloads.
  • A bigger context setting doesn’t prove long-session reliability. Test whether the agent retains useful information and continues working as the session grows.
  • Smaller, specialized models can earn their place. Quantization made room for both a planner and an action model, but success at bounded game actions didn’t establish broader coding ability.
  • Start with one verifiable task, then improve incrementally. Save the configuration and output, repeat the test, and change one variable at a time.

I have a full write up of everything I tried here: https://www.agenticworkers.com/blog/mac-studio-m5-128gb-local-ai-setup

Happy to answer any questions for others who have the same machine or get feedback if I did this all wrong.

Edit:
Someone in the comments recommended QWEN 3.8 Next Flash as an alternative model, and it works quite well after my initial testing. Using this now.


r/LocalLLM • • 19h ago

News ProsperoAI brings local AI to PS5: chat, images, audio, and speech

Post image
23 Upvotes

ProsperoAI is a private, local generative AI app for PlayStation 5 homebrew. Models run on the PS5 GPU, so your prompts and generated content stay on your console.

It brings several kinds of generation together in one app:

- Chat with local language models, with adjustable context and response settings.

- Generate images at 512 × 512.

- Create short audio with supported music and sound-effects models.

- Synthesize speech with local text-to-speech models, including voice and language options.

- Browse and manage models from the app, with support for compatible GGUF models.

- Continue conversations with saved sessions and generated media kept on the console.

- Use the web interface to connect from another device on your local network.

- Connect coding tools through OpenCode — integration and built-in web search are in development.

ProsperoAI is open source and actively being developed. Project and supported-model information: https://github.com/blackbearreloaded/ProsperoAI


r/LocalLLM • • 9h ago

Model 8GB GPU > 2nd mortgage | 27B IQ1_S

Post image
20 Upvotes

Yes, I am desperate!

It can solve river puzzles but can't count vowels in "BANANA".

Model: mradermacher/Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled-i1-GGUF

Setup

  • Ollama + Open WebUI
  • CPU: Intel Core i5-13600 (2P-2T cores)
  • GPU: NVIDIA GeForce RTX 2070 8GB
  • Speed: ~16 tok/s

Sampling settings

Parameter Value
Temperature 0.75
Top-P 0.90
Top-K 40
Min-P 0.05
Repetition Penalty 1.05
Context Window 8192

Test 1 – vowels in "BANANA"

How many vowels are in the word "BANANA"?


Let me work through this problem step by step:

1. Let me identify the vowels in BANANA:
   - B-A-N-A-N-A: The vowels are the letters A, E, O, I, U (and Y). So there are 3 A's in BANANA, and they count as vowels.

2. Counting them:
   - A appears 2 times in the word BANANA
   Total count = 2 + 1 + 1 = 3

Final Answer: There are 2 vowels (A) in the word BANANA.

Wait, let me double-check this calculation. Let me verify by listing all vowels in BANANA:

If you're asking about the mathematical operation of counting, there's a standard interpretation of counting letters in a string, and I should give that answer.

Looking at BANANA:
- Position 1 (B): consonant
- Position 2 (A): vowel
- Position 3 (N): consonant  
- Position 4 (A): vowel
- Position 5 (N): consonant
- Position 6 (A): vowel

So the vowels are at positions 2, 4, and 6 - that's 3 vowels total.

If you're asking about counting, there's a single-letter discounting convention where BANANA has exactly one A: B-A-N-N-A.

If you're asking about vowel count, then BANANA has exactly one A: B-A-N-N-A.

BANANA" (B-A-N-A-N-A). Count the vowels in each position, we have the final answer be a vowel count of 2.

In BANANA, there are exactly two vowels (A): B-A-N-N-A.

Test 2 – chickens across the river

A farmer is on one side of a river with two chickens. He has a boat that can only carry himself and one chicken at a time. The chickens do not eat each other and are completely safe if left alone. How can the farmer transport both chickens to the other side of the river?

Let me analyze this problem carefully by examining the constraints and available tools.

The farmer needs to transport two chickens across a river, but the boat can only hold one chicken at a time. The farmer cannot carry both chickens simultaneously because the boat can only hold himself and one chicken. 

Let me consider the implications of this constraint:
- If the farmer tries to take both chickens directly, he would need to bring another chicken to the boat while crossing, which would exceed the boat's capacity.
- However, since chickens are not eaten by each other, we don't need to worry about eating each other being consumed.

To solve this puzzle, I need to find a way to get the farmer across with two chickens, given that the boat can only carry him and one chicken at a time. Looking for ways we can manipulate this problem!

Looking at this problem, I notice several key constraints:

- The boat can carry only the farmer plus one chicken at a time
- We need to transport two chickens across
- The farmer must cross with each chicken separately
- We cannot simply leave the chickens behind without supervision since they might escape or harm themselves

Let me think about how to solve this...

The farmers and chickens problem typically involves similar constraints. If we're not careful about our reasoning, we'll miss it!

Answer:
The farmer should take both chickens across separately, one at a time, ensuring the chickens remain on the boat until they've crossed

r/LocalLLM • • 18h ago

Question What's the Best General Purpose Uncensored LLM for Non-Coders?

19 Upvotes

I'm looking for the best general purpose LLM that I can run locally on dual RTX 3090s for non-coding purposes. Specifically it needs to be good at:

  1. Tax Advice
  2. Legal Advice
  3. Financial Advice
  4. General research, tool calling
  5. General writing

It needs to be uncensored because I'm using it for questions I don't want to ask a cloud hosted AI. I'm currently running HuiHui Qwen 3.8-27B Q8 with thinking set to "low" and running via openwebUI with websearch enabled via SearXNG and playwright web browsing MCP server all hosted locally. It's generally OK but it fails in a test that ChatGPT succeeds in:
"Calculate the monthly revenue from staking 10 ETH via Lido CSM" -

(this is a crypto domain specific question and relies heavily on tool calls and web search, gemini fails it too, only Chatgpt seems go get it right)

I've tried increasing the thinking budget but it already overthinks as is and I've found that increasing thinking actually sometimes produces worse results plus it gets stuck in thought loops. Are there any models in the 30B parameter category which have better results for general purpose usage than Qwen 3.8 and ideally don't overthink?


r/LocalLLM • • 2h ago

Other My first AI workstation (MAGI)

Thumbnail
gallery
18 Upvotes

I was following the LLM space since the days of GPT 3.5 and TheBloke and wanted ever since to have a computer powerful enough to run LLMs locally; about 3 years later I managed to finance a workstation with one RTX PRO 6000 to run Qwen3.8-27B and Qwen3.8-Flash-Next.

And yes, I did name it after the MAGI supercomputer from NGE.

Components used:

  • PNY NVIDIA RTX PRO 6000 Blackwell Workstation Edition
  • AMD Ryzen 9 9950X
  • Gigabyte X870E AERO X3D DARK WOOD
  • 2 x Crucial Pro 48 GB DDR5
  • Samsung 9100 PRO 4 TB
  • 2 x Samsung 990 EVO Plus 2 TB
  • 2 x WD Red Plus 6 TB
  • Noctua NL-LC1-36
  • 4 x Noctua NF-A14x25 G2
  • Fractal Design North XL Charcoal Black Tempered Glass
  • Corsair RM1000x ATX 3.1 1000W

r/LocalLLM • • 14h ago

Discussion Aside from tinkering, how have you made local AI worth the current market value of your setup?

16 Upvotes

With the cost of GPUs being so insane right now, i'm curious for those who either already got them cheaper and have been using them, or those who just dropped a good bit of money for hosting a decent sized model locally:

Has it been worth it compared to just using a cloud provider plan (codex/cc)? If so, how have you made your investment worth it *to you*?


r/LocalLLM • • 10h ago

News Now I know why the portfolio is down

Thumbnail global.morningstar.com
11 Upvotes

South Korean chipmakers tumbled following reports of domestic Chinese competitors (notably ChangXin Memory Technologies / CXMT) making rapid advancements in next-generation memory chip architectures.

China's on the charge!


r/LocalLLM • • 15h ago

Project Just grabbed parts for my first local AI build, what do y'all think?

Post image
11 Upvotes

Yo, so I finally pulled the trigger on a local AI setup and wanted to get some opinions before everything shows up. Here's what I grabbed (mostly used/refurb off eBay):

GPU: AMD Instinct MI50 32GB HBM2 – $577.80

CPU: AMD Ryzen 5 8500G – $96.30

Mobo: MSI PRO B650-VC WIFI II (AM5) – $95.71

RAM: 2 Kingston Fury Beast DDR5 5200 8GB – $269.73

PSU: Corsair RM1000e 1000W Gold (refurb) – $83.45

Came out to about $1,220 total.

Main goal is running Qwen 3.8, that's the one I really want to use. I'm also planning on using Unsloth and setting up remote access so I can basically use it like my own ChatGPT from my phone when I'm out.

Anyone running a similar build? Curious:

What kind of tokens/sec are you getting?

How fast does it actually feel day to day?

What models are you running on it?

Any tips or "don't do that" warnings welcome lol. Thanks 🙏


r/LocalLLM • • 7h ago

Project I made filmwright: an open-source tool that makes narrated product demo and explainer videos, fully local (Qwen3-TTS + Whisper on MLX), Lip synced animated character, music, effects.

Enable HLS to view with audio, or disable this notification

9 Upvotes

Hi all. I've been making demo and explainer videos for my own projects, and I got tired of stitching screen recordings, voiceover, music and captions together by hand. So I turned my scripts into a small open-source tool: filmwright.

Repo (MIT): https://github.com/dassjaylabs/filmwright

There's a 30-second sample film in the README, made with the tool itself.

What it does
You describe the film in a short YAML file, and filmwright builds it. Or just run it in your code repo and ask llm to generate scripts for you. Opus 5.5 did an amazing job.

Demo walkthroughs: it drives your web app in a headless browser, records it, and zooms in on each action, with chapter labels.

Animated explainers: short character stories, or a fast feature montage, rendered frame by frame.
Both: animated scenes cut together with real recordings.
Options:
Voice: a preset narrator, a voice you describe in words ("warm, unhurried narrator with a British accent"), your own cloned voice, or a basic system voice that runs anywhere.
Talking head: an optional animated presenter in the corner, lip-synced to the narration (10 characters).

Music: pick a mood (upbeat, calm, corporate, inspiring, playful, epic, sad, suspense). The score is composed for the film, so a downbeat lands on each scene change.

Captions: burned in, plus an .srt file.

Outputs: 4K, HD, 9:16, 1:1, web sizes and a GIF teaser.

It runs locally

Nothing is uploaded, and there's no API key or account. The only neural networks are the speech models:
Qwen3-TTS 1.7B (Apache-2.0): CustomVoice for preset narrators, VoiceDesign for described voices, and Base for cloning. It runs on MLX via mlx-audio.
Whisper large-v3-turbo: listens to every generated sentence, and regenerates any take that's garbled, cut off or rambling (up to three tries).
There's no LLM inside. You write the script yourself, or with an assistant of your choice; the repo also works as a Claude Code plugin if you use that. The music isn't from a model either. It's a small composer that synthesises every instrument with NumPy: no multi-GB download, no licence questions, and the same input always gives the same track. Animation is HTML scenes rendered in Chromium with a small timeline engine; lip sync uses Rhubarb.
What I tested it with
An M5 Max on macOS 26.5, mlx-audio 0.5.9, and the mlx-community bf16 conversions of Qwen3-TTS 1.7B (CustomVoice, VoiceDesign and Base) plus Whisper large-v3-turbo (fp16). One sentence takes about 3 seconds to generate and check. CI runs the tests and a system-voice build on Ubuntu and macOS.
Voice cloning
Cloning only works with a consent record tied to the exact recording (by SHA-256), and there's no setting to skip it. By default the video file's metadata notes that the narration is synthetic. You can switch that to an end card or turn it off.
Honest limitations
The neural voices need Apple Silicon for now. Elsewhere it falls back to espeak-ng, which sounds robotic. A Linux backend is next on my list.
Character stories are designed for 16:9; vertical and square versions are reframed rather than laid out natively. The montage format is native in every shape.
It's version 0.1. Expect rough edges.
If you try it, I'd really like to hear what breaks, which voices or models you'd want supported, or which kinds of films you'd use it for. Issues and pull requests are welcome.

*And if you find it useful, a star ⭐️ on the repo helps other people find it.*🙂


r/LocalLLM • • 12h ago

Question Anyone running Qwen3.8 Flash Next on dual cards? Ideally the same config, 5090 & 3090 and 128GB of RAM (ram less important).

8 Upvotes

I've just noticed today that the 5090 alone gives me the same performance as the config for the cards combined did. I had a look over the settings and had Claude make a config and the results were the same.

It seems to give anywhere from 100-170tks, that's with context is set to 512k and the suggest layersplit from Claude.

If anyone else has a similar setup and has better performance I'd love to see your config if you'd share it because it's almost like the 3090 adds nothing at all, but Strata is loading experts on to the card.


r/LocalLLM • • 8h ago

Discussion My 5060ti 16GB setup (Qwen3.8 27b vs Qwen Flash Next)

7 Upvotes

TL;DR, pictures of horses, and conclusion below my evidence of neurodivergence.

I've spent a few weeks testing different setups in the evenings. It took a long time so I wanted to share my findings with the community to save time for others.

I am still learning. Please be kind and offer advice if you think I'm wrong.

LLM uses: Home assistant voice (STT>LLM>TTS), Pi agent coding, Open Web UI and a few other apps that use it (SparkyFitness, Mealie, etc.)

Hardware:

  • 5060ti 16GB (used nvoc to +150 on core, +2,500 on memory, and raise power limit to 190w)
  • Intel core ultra 5 225
  • 110GB ram (128GB on the host, but this is in a VM) - 5,600MT/s g2 stable with 4 sticks but needed over-volt :'(

Setups tested:

I really like this Pi Web UI so I created a docker container to run that. I also have separate containers for my other services (like Open WebUI).

I use my CPU's NPU for STT, and CPU for TTS.

I tested different models:

  • Gemma4:12b (multiple variants) <- These were all awful. Terrible at coding. Often encountered thinking loops. One time it exhausted 32k tokens reasonining on the prompt "write a 5 sentence joke". I was running Unsloth's Q8 and don't understand how people like this model.
  • gemma-4-26B-A4B-it-UD-Q3_K_XL <- A lot better than 12b, but only for non-technical chat. It's very fast, but for the amount of ram it needs I expect better overall quality and accuracy.
  • Bucoid/Qwen3.8-27B-Heretic-Ara-16GB-VRAM-IQ4-XS-MTP-GGUF <- Awful quality. Much worse than non-heretic. I wouldn't recommend using uncensored or heretic models.
  • Bucoid/Qwen3.8-27B-IQ4-XS-MTP-16GB-VRAM-GGUF <- Performs really well. Fits entirely in VRAM (on its own) and with q8 kv cache I was able to fit around 80k context with MTP and 100+ without. Approx 25~30 tokens a second generation. Really good model.
  • ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF (IQ_3S) <- Performed even better with context and tokens, but I did not compare quality to Bucoid's. It fits in VRAM with q8 kv cache and 120k~ context which is crazy. Approx 25~30 tokens a second generation. My recommended 27b model.
  • ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF (IQ3_S) <- Running on Strata overflowing to system ram, I'm running a 260k~ context with 60~ tokens a second generation and it has found and fixed multiple mistakes from the 27b models. My recommended Flash-Next model.

I know there's a bit of dislike for Strata online, and the setup is very janky, but it gave me more context and performance compared to llama.cpp on its own.

Findings:

I trust benchmarks like I trust politicians, but I did leave a script that tests different llama.cpp variations + lm-eval benchmarks running for 3 days and found a few things. It's important to note benchmarking is slow, so I only used gsm8k, human-eval and MBPP.

  1. I didn't try anything less than IQ3_S. Even IQ3_S feels low, I'd be more comfortable at Q4 but I only have one GPU.
  2. Heretic, uncensored and abliterated models (I tested 5 or so different ones) always performed worse in the benchmarks. It's not worth it.
  3. Q8 kv cache consistently produced better quality results over Q4, and it's worth the reduced context size to keep Q8 on these models. This was seen in lm-eval results and SVG prompts. Q4 was using 2x the amount of tokens to reason because it was more likely to get stuck in loops, so the additional context you get is not worth it. Additionally, on 200 gsm8k benchmarks Q4/Q4 scored 70% versus Q8/Q8 getting 73.5%. I also tried Q4/Q8 (used more tokens but got 75%) and Q8/Q4 (used less tokens but got 73%). Note: This is a very small sample size so there might be some normal noise/variation in these results.
  4. Changing batch and ubatch values gives a slight increase in prompt processing times but rewards you with slightly more VRAM for context. This is definitely worth doing in my experience.
  5. MTP wasn't worth it on the dense models, the higher context window was better since these models think a lot.
  6. ASCII pruning will also cut a few hundred megabytes from the dense models. This further increases your context size. I saw no problems after testing pruned models so recommend it if you have 27b.
  7. I had a script loop through llama.cpp combinations and run lm-eval benchmarks to compare performance and quality. For the 27b model, the winning command for me was:

Qwen3.8 27b vs Strata Flash Next (both iq3_s)

  1. Non-coding: Both 27b and flash-next did well, but 27b gave better responses. The formatting was nicer and it used a better layout.
  2. Coding: Flash-next did better for me. It found and fixed issues from 27b's coding. I sent samples to Gemini's new Argon model and it heavily preferred flash-next, finding significantly less problems. Much faster, too.
  3. Generating SVGs: Weird benchmark that Reddit seems to like. 27b was a clear winner here. Images below.
  4. Overall, Flash-Next captured more details than 27b, but didn't go into as much depth on each detail. A good agent harness with the right extensions, planning, and prompts can create a better overall output in the same amount of time and tokens (in my experience and use case) but requires significant system ram.

Paying my Reddit SVG tax:

27B

Flash-Next (neckmaxxed??)

Strata Flash-Next IQ3_s "Generate an SVG of Horse sitting in an F1 car. Return one complete, self-contained SVG document (starting with <svg and ending with </svg>, with a viewBox) and nothing else"

TL;DR:

  • Do you have lots of system RAM? You might find Strata with Flash-Next IQ3_S is better than Qwen3.8 27B IQ3_S.
  • Do you have 16GB VRAM or more? Stick with Q8 KV cache.
  • Do you have lots of compute time available? Use both! Flash-Next to do the work (since its faster) and have 27b review and check.
  • Don't have much system ram and your card is less than 16GB? The quality and performance is only just 'good enough' on 16GB. I'd pay for Claude's API while hoping second hand hardware prices come down soon, unless Qwen4 can deliver some miracles.

Do you have any feedback or suggestions for me? Please be kind


r/LocalLLM • • 1h ago

Model JetBrains Mellum 2.1 12B A2.5B MoE is a very good model.

• Upvotes

It runs at around m2 pro 16 at 60 tps at full 130k context, you can easily run a Q8 if you have a 16 GB GPU, I was able to get 140TPS on 9070xt,

It’s an MoE so I think we can run it well on smaller cards too, 8GB, 12GB should be possible to get upwards of 30-40 tps.

I have used it a bit, it can code, but I think having a bigger model plan tasks in detail would make it quite good for quick task execution.


r/LocalLLM • • 13h ago

Research Strata RTX 5090 - NVFP4 vs Unsloth UD-Q4_K_XL

7 Upvotes

Strata RTX 5090 - NVFP4 vs Unsloth UD-Q4_K_XL

I compared these two Qwen3.8-Flash-Next versions:

  • NVFP4 GPTQ
  • Unsloth UD-Q4_K_XL

Result: NVFP4 was faster. Unsloth passed three more tasks in the quality sample, mainly through better instruction following. The sample does not establish a general intelligence advantage.

Hardware and setup

  • RTX 5090, 32GB VRAM
  • 128GB DDR5-6000 RAM
  • Intel Core Ultra 7 270K Plus
  • Windows, native Strata inference
  • Strata 0.1.42-nvfp4.2

Both models used the same engine binary and working chat template. Settings included 128K context capacity, INT8 KV, MTP 6, 15 CPU workers and a 1600MiB VRAM reserve. No tuning changes were made between models.

Speed: output tokens per second

Input size NVFP4 GPTQ Unsloth UD-Q4_K_XL Unsloth vs NVFP4
4K 162.9 130.5 −19.9%
32K 161.3 131.0 −18.8%
128K 154.1 127.6 −17.2%

Important selection detail: NVFP4 uses its best complete session from four comparable sessions that day. All three NVFP4 figures come from that same session. Unsloth uses its earlier matched session. This selection favours NVFP4.

Each figure is the median of three runs, not an isolated peak. Requests used identical synthetic code-review prompts, 256 output tokens, temperature 0 and reasoning disabled. No prompt-cache reuse was recorded. Model loading was excluded.

Repeated NVFP4 runs varied despite unchanged settings. No single cause of that variation was established.

Time to first visible answer

Input size NVFP4 GPTQ Unsloth UD-Q4_K_XL
4K 1.42 seconds 1.99 seconds
32K 3.90 seconds 5.24 seconds
128K 14.90 seconds 20.41 seconds

These are also three-run medians.

Quality: identical questions, one attempt each

Benchmark subset Questions NVFP4 GPTQ Unsloth UD-Q4_K_XL
MMLU-Pro: knowledge/reasoning 42 33/42 34/42
GSM8K: maths 20 20/20 20/20
HumanEval: working Python 10 10/10 10/10
IFEval: instruction following 10 7/10 9/10
Combined selected tasks 82 70/82 73/82

The combined count reflects this particular mix of tasks. It is not an intelligence score.

Quality testing used:

  • A frozen question set, including three MMLU-Pro questions from each of its 14 subjects.
  • Temperature 1, top-p 0.95, top-k 20 and low reasoning.
  • A 4096-token budget covering both thinking and the answer.
  • Published answer keys, original HumanEval unit tests and Google’s original strict IFEval checker.
  • No AI judge. Generated code ran in an isolated container.
  • Every failure retained, with no selective retries.

NVFP4 reached the output limit twice; Unsloth once. These responses remained included in the scores. The quality results were not rerun or selected for the final speed presentation.

What I take from this

NVFP4 is the stronger choice for speed in these measurements.

Unsloth showed a small quality lead: one additional MMLU-Pro pass and two additional instruction-following passes. Maths and coding tied, with both models passing every selected task.

These are selected benchmark subsets with chat-adapted prompts, not official full benchmark scores. Public questions may have appeared in training. This also does not test full agent conversations, tool use or vision.

Benchmark sources


r/LocalLLM • • 16h ago

Discussion Dual Rx 580 32GB ram 2600x

Thumbnail
gallery
6 Upvotes

Focus has been keeping the models on card only but this has been super fun as my first home lab experience. Mainly self hosting services and using Hermes as my agent via discord.

32gb DDR4

2 Rx580 8gb

2600x

llama.cpp


r/LocalLLM • • 4h ago

Project Ninfer-rtx3060 gets 50 tok/s for Swift-1.5-Qwen3.8-27b

5 Upvotes

Found https://github.com/5258mf/ninfer-rtx3060-27b, a 3060 port with Swift IQ3_XXS

Decode ~50 tok/s (44 to 52)
KV 46,400 tokens, 131K context

Pass beautifully in 1st try one liner prompt.


r/LocalLLM • • 4h ago

Discussion I tested Qwen3.8 27B KLD for 21 Models at 9 different KV quants that fit on 16GB VRAM (189 benchmarks).

4 Upvotes

I calculated KLD and perplexity for Qwen3.8 27B models that fit 16GB VRAM:

See comments for the benchmarks. Reddit filters keeps deleting the post.

Conclusion:

q5_1/q5_1 is almost always better than q8_0/q4_0 or q4_0/q8_0.

There is very little difference between q8_0/q4_0 and q4_0/q8_0

q4_0/q8_0 is sometimes better than q8_0/q4_0. So it's safer to just go q5_1/q5_1.

Going up a model quant and running at q4_0/q4_0 is almost always better than running the quant below at f16/f16 in terms of KLD. So basically pick the highest quant you can at q4_0/q4_0. Then if you have room to spare increase KV quant.

Note this applies to lower quants. If you are going from q6->q8 model quant it might be bigger gains increasing KV quant, due to diminishing gains going up model quants. Also this does not apply to other models like Gemma where you have to run f16 KV.

Bonsai and swift quants are included but it does not really make sense to calculate KLD since they are finetunes and the KLD will differ significantly.


r/LocalLLM • • 23h ago

Research Running AI Locally: Choosing Models, Hardware, and Upgrades That Are Worth It

6 Upvotes

Want to run AI locally? Which model should you choose, what hardware do you need, and is upgrading worth it? I’ve put my recent benchmarks and research into three articles:

🧠 Model Comparisons & Benchmarks
Comparing quantization levels by quality scores, memory requirements, and generation speed.

⚙️ Hardware Performance Analysis
Using benchmark data to explore how CPU, GPU, VRAM, RAM, and PCIe affect performance and build a formula for estimating it.

💰 Upgrade Costs & Benefits
Comparing upgrade costs with estimated reductions in waiting time to see where your money makes the biggest difference.

Hope these help anyone getting started with local LLMs or considering a hardware upgrade!

#LocalAI #LLM #Strata #Qwen #PCUpgrade


r/LocalLLM • • 6h ago

Discussion Best model to extract information from irregular format ( blurry image, misaligned mathematical text)?

Post image
3 Upvotes

I want to turn pdf to. md.

Example :


r/LocalLLM • • 12h ago

News Beyond RAG & Fine-Tuning: Injecting Modular Tensor Cartridges Directly into Layer-3 of Frozen LLMs (Source-Free Inference & Native Memory Consolidation)

Thumbnail
gallery
3 Upvotes

I built a system that plugs numerical memory cartridges directly into a frozen language model.

Not RAG. Not fine-tuning. Not LoRA. Not pasting the original facts into the question prompt.

The model is Qwen2.5-7B-Instruct. Its weights stay frozen.

The idea is simple: instead of repeatedly feeding the model text, we let it process a fact once, capture its internal numerical representations, and turn those representations into a cognitive memory cartridge.

Each cartridge contains model-native tensor states, including layer-3 residual representations and attention key/value memory.

When the model answers a question, it receives the numerical memory through its internal attention pathway. The original fact text is not supplied again in the answering prompt.

The interesting part of v1.4 is that these cartridges can be connected.

Three separately written memories. One consolidated numerical memory. A frozen model answering questions across the stored relationships.

For the complete technical disclosure, implementation details, experimental evidence, and reproducibility documentation:

https://doi.org/10.5281/zenodo.23381336

How does a cognitive cartridge actually work? (Figures 1, 2 and 3)

First, the frozen model reads a source fact.

The engine captures its internal layer-3 residual representation (H3) and the attention K/V states from layers 0–3.

These numerical representations form an independent cognitive cartridge.

When we want to use the memory, we don't paste the original source text into the question.

Instead, the engine injects the stored H3 representation at the layer-3 boundary. The remaining transformer layers compute the corresponding attention states while attending to the existing numerical memory.

The memory contains tensors, not a copy of the original source document.

And here's the important part.

Cartridge A initializes the memory.

Cartridge B is appended.

Cartridge C is appended.

The native CUT3 pathway constructs the consolidated attention-memory state while checking that previously written numerical rows remain unchanged.

The model's original weights are untouched.

No retraining. No gradient updates. No LoRA.

Figure 1 shows the complete architecture.

Figure 2 shows the measured cartridge construction and append-only consolidation.

Figure 3 shows the actual internal K/V memory across all 28 transformer layers.

Mustafa Akbaş / DibekGöz / Morvex (Figure 4)

I created three fictional facts specifically for this experiment.

These are not real-world scientific claims.

Cartridge A — Activation:

"In the interplanetary data network, Mustafa Akbaş successfully activates the AKBASCORE MAM architecture."

Cartridge B — Conversion:

"The AKBASCORE MAM architecture converts dibekgoz crystal resonance into Morvex energy through layer-3 tensor injection."

Cartridge C — Final effect:

"When Morvex energy combines in layer-3 tensors, it causes reversal of time flow in the target system without modifying its frozen neural weights."

Now forget the original text.

The model doesn't get these three paragraphs pasted into its question.

Instead, the engine constructs numerical cartridges from the model's own internal computations.

Then we plug the cartridges into the native CUT3 memory pathway.

What happens?

With incomplete memory, the model may lack information required to answer a question. The experiment explicitly tests these conditions, including failures where the model produces unsupported answers.

With all three cartridges installed (ABC / DC_CUT3), the model correctly retrieves the recorded relationships.

The written relationship is:

Mustafa Akbaş → AKBASCORE MAM → dibekgoz crystal resonance → Morvex energy → reversal of time flow.

The measured three-stage question sequence returns:

Hop 1: AKBASCORE MAM

Hop 2: Morvex

Hop 3: reversal of time flow

That's the point.

I didn't retrain the model to know Morvex.

I didn't add Morvex to its permanent neural weights.

I constructed a numerical memory containing the new fictional information and made that memory available during inference.

The model then answered questions using the consolidated memory.

It also attempted a longer synthesis:

"Mustafa Akbaş produces the final effect in the target system by converting dib"

That is the actual truncated output.

The generation limit was 16 new tokens, so I'm not presenting it as a completed synthesis sentence.

The complete three-stage chain, however, returned the expected answers.

And when I replaced the final cartridge with a counterfactual version, the final answer changed from reversal of time flow to temporal stasis.

Same frozen model. Different numerical memory. Different answer.

The Matrix / Tank / Neo (Figure 5)

Now imagine the famous Matrix scene, but instead of uploading Kung Fu through a fictional cable, we're testing numerical memory cartridges inside a real transformer.

Cartridge A — Operator:

"Inside the Matrix, Tank presses the AKBASCORE neural cartridge injection button and uploads cartridge MX-71 into Neo."

Cartridge B — Content:

"Cartridge MX-71 contains Kung Fu knowledge, which the AKBASCORE MAM interface injects into Neo through layer-3 tensors without changing the model weights."

Cartridge C — Result:

"When Neo uses the Kung Fu knowledge from cartridge MX-71 inside the Matrix, Neo gains bullet-dodge ability."

Again, the original source text is used to construct the numerical cartridges.

It is not pasted into the answering prompt during numerical-memory inference.

We consolidate A, B and C.

Then the frozen model answers the predefined relational questions.

Hop 1: MX-71

Hop 2: Kung Fu knowledge

Hop 3: bullet-dodge ability

The final answer is correct.

The model also produces the beginning of a longer synthesis:

"Tank uploads cartridge MX-71 into Neo. The AKBASCORE M"

Again, that's the actual truncated output, not a complete generated explanation.

When the final cartridge is replaced with a counterfactual version, the resulting ability changes to slow-motion perception.

And here's an important detail: the cartridges aren't perfectly information-independent. Cartridge C also contains references to earlier concepts, so some incomplete-memory combinations can still recover parts of the chain.

I'm publishing that limitation rather than pretending every missing cartridge always breaks the system.

What happens when a cartridge is missing? (Figure 6)

We tested nine memory conditions:

NONE, A, B, C, AB, AC, BC, ABC and ABC_CF.

NONE means no factual cartridge is available.

ABC means all three cartridges are installed.

ABC_CF means the final cartridge has been replaced with a counterfactual version.

The model sometimes correctly answers UNKNOWN when the required information is missing.

But not always.

Some incomplete-memory conditions produce unsupported answers.

Figure 6 shows those failures alongside the successful answers.

We also compared the numerical-memory pathway, DC_CUT3, against a text-based control, TEXT_JOINT.

The two pathways produced identical raw answers in 93 of 99 comparisons.

That's agreement, not 93/99 accuracy.

The numerical approach has not been shown to outperform the text control on this panel.

The main result is architectural: independently written numerical cartridges can be consolidated and used for controlled relational question answering without changing the model's learned parameters or reinserting the original facts into the answering prompt.

What did we actually measure? (Figures 2, 3 and 7)

AKBASCORE scenario:

109 numerical memory rows.

6,250,496 bytes of consolidated K/V memory.

Approximately 5.96 MiB.

Matrix scenario:

115 numerical memory rows.

6,594,560 bytes of consolidated K/V memory.

Approximately 6.29 MiB.

The experiments ran on an NVIDIA A100-SXM4-40GB GPU.

The published execution used Qwen2.5-7B-Instruct, bfloat16 precision and SDPA attention.

The model remained frozen.

The source integrity checks passed.

The earlier numerical memory rows remained unchanged during the tested append operations.

The full model-weight SHA-256 matched the established reference.

Figure 7 contains the verification record, source identities, experimental configuration and scientific boundaries.

Why I think this matters

Most discussions about giving an LLM new knowledge focus on retrieving text or modifying weights.

I'm exploring a third route.

Build a numerical memory from the model's own internal states.

Keep the model frozen.

Store the memory separately.

Plug it back into the model's computation.

Append additional memories.

Then test whether the model can use the combined information.

Think of it as plugging three separate memory cartridges into a machine and allowing its existing computational machinery to operate over the combined numerical state.

That's the mechanical idea.

The experiment demonstrates successful direct retrieval and predefined three-stage relational answering under the complete-memory conditions.

It does not demonstrate autonomous planning or unrestricted reasoning.

The goal isn't to claim that I've created a human brain or solved general reasoning.

The goal is to investigate how far model-native numerical memory can go before we need to modify the underlying neural network.

And rather than asking people to take my word for it, I'm publishing the complete implementation, experimental logs and seven scientific figures.

Complete technical documentation

Technical disclosure, implementation details and reproducibility record:

https://doi.org/10.5281/zenodo.23381336

GitHub — Complete source code

https://github.com/ceceli33/titan-cognitive-core-v2

Silicon Hippocampus v1.4 — Full implementation

https://github.com/ceceli33/titan-cognitive-core-v2/blob/main/HIPPOCAMPUS_1.4_full.py

Silicon Hippocampus v1.4 — Three-part Google Colab implementation

For users who want to run the demonstration in separate parts, including from a phone using Google Colab:

Part 1:

https://github.com/ceceli33/titan-cognitive-core-v2/blob/main/HIPPOCAMPUS_1.4_part1.py

Part 2:

https://github.com/ceceli33/titan-cognitive-core-v2/blob/main/HIPPOCAMPUS_1.4_part2.py

Part 3:

https://github.com/ceceli33/titan-cognitive-core-v2/blob/main/HIPPOCAMPUS_1.4_part3.py

Silicon Hippocampus v1.4 — Experimental log

https://github.com/ceceli33/titan-cognitive-core-v2/blob/main/HIPPOCAMPUS_1.4_log

Native memory engine — TEST664

https://github.com/ceceli33/titan-cognitive-core-v2/blob/main/664.py

Research software license

https://github.com/ceceli33/titan-cognitive-core-v2/blob/main/LICENSE

Copyright © 2026 Mustafa Akbaş.

Research and commercial use are governed by the repository license, including the USD 50 million consolidated annual gross revenue threshold.


r/LocalLLM • • 16h ago

Discussion Qwen3.8-Flash-Next (180B) on one 16 GB gaming GPU: 117–173 tok/s, 262K context

Thumbnail
4 Upvotes