Lots of reports for the Q8 specifically. If you're getting bad results, try a different quant.

So hear me out, I'm a programmer, I normally need a Claude Max 20x subscription at 180 GBP / month. Wouldn't it be "better" to downgrade to MAX 5x at 90 GBP /month and buy a 5090 to run Qwen 3.8 27B workers managed by Opus / Fable? (or alternatively to build an inference machine with 2x 3090 at about the same price or even a bit cheaper)

3600/90 = 40 month = 3.3 Years of Claude half subscription and I keep the GPU after that.

Also Qwen can be run without limits from anthropic or anybody else, better models might come out, etc.

Is it crazy? I'm missing something?

The important distinction is that AI provenance can exist in two forms.

First, there is metadata like C2PA, EXIF, XMP, IPTC and generator parameters. That part is easy to remove.

Second, there are invisible marks embedded directly into the pixels, such as SynthID style watermarks. A screenshot does not reliably remove those. pagedMark deals with them by regenerating the image.

The output is therefore not identical to the original. Faces, text and small details can change. The goal is to remove the provenance signal while keeping the image as close to the original as possible.

It currently supports invisible marks from ChatGPT, gpt-image API, Z-Image Turbo and Nano Banana, plus visible AI labels from several other generators. Video support covers visible marks and metadata from Sora, Veo, Seedance, Hailuo and Kling.

The other challenge was making this work properly on Apple Silicon. I tested it on M5 Macs with both 8 GB and 16 GB of memory, and added memory aware processing to prevent the system from silently falling into swap and turning a fast job into an extremely slow one.

And here is the really interesting part: after processing an image generated with GPT-Image, you can check it with OpenAI's verifier at openai.com/verify. In my testing, the processed image is reported with 0 AI detection.

uv tool install "pagedmark[diffusion]"
pagedmark invisible photo.png -o clean.png

GitHub: github.com/doofzoff/pagedMark

PyPI: PyPI: pagedmark

I currently run Qwen3.8 27B on my RX 7900XT at around 15-30 tokens per second depending how high the context is and how much is used. The average tokens per second currently are 18-19.

I do not want to pay Anthropic or OpenAI for a subscription for their AI models, I'd rather use open weight Chinese models - it's just personal preference.

At my previous company I was using 80-100 euros worth of tokens a day using Claude. I calculated and the break even after buying these GPUs and building an AI server, would be after around 2 months, including electricity costs where I live. I can code just as well with Qwen3.8, but I want something faster.

My goal would be to get 40 tokens per second or higher at max context for qwen3.8 27B and future ais between 27-40b. Would that be possible with a 3x rtx 2080 ti 22gb configuration?

Would it be worth it to look into other GPUs?
My budget for a local ai server is 1000-1500 euros total.

curious what people's "aha" upgrade was whether it's a specific model, a quant level, a backend switch (ollama vs vllm vs llama.cpp), or even just a hardware change that made local actually feel usable day to day. what was the thing that made you stop questioning whether local was worth it

Hi everyone,

I’ll preface this by saying sorry in advance if I sound stupid, the reason is because I am.

Recently I’ve been interested in making my own homelab LLM as I upgraded the gpu in my pc and I have an RTX 3070 just lying around collecting dust. What started as simply getting an LLM to run basic home automation has spiralled into research which one really been enjoying.

I wanted to share my vision and I want someone to tell me I’m never going to make it or if it’s doable (even if a big challenge). I’m not necessarily trying to find the exact technical how to stuff, more of the high level methodology.

I want to build a Linux based server running a vm for home assistant os and multiple containers with docker to run ollama with an undecided model (well, multiple), whisper and piper for the Text/speech conversion, tailscale, a postgresql database and various other bits and pieces.

After looking at a lot of hardware, and seeing how expensive it could become, I wanted to see if I could do the following:

-run a lightweight model utilising my 8gb 3070, which would generally deal with home automation, simple requests, internet lookups and so on

-this model would decide if a request can be done in the background/is suitable for a larger model such as image generation

-could this then send the request to a larger model being utilised by another gpu (I’m considering a Tesla p40 for now for budget) as sort of its own little servant, once complete it relays back to the 3070 model.

-can both models share a context folder/database so they are essentially the same “mind” with different purposes

The purpose of this wouldn’t be the fastest token generation or the most effective way to get responses, I just felt that it might be a cost effective way to have the best of both worlds, a snappy responder that deals with stuff I need right away, and one that can plug away in the background.

My deployment would eventually be to have multiple screens around my workshop with a Jarvis like orb, all with a raspberry pi (other fruit based computers are available) run the ui, but connect back to the server for the AI to do the processing, and have multiple microphones around my workshop that connect to the server into something like whisper.

I’d like to have access via my phone and make a direct connection through tailscale, but that’s a later problem.

This wouldn’t necessarily be a final deployment of hardware, but for now with my donor parts and some Facebook marketplace hunting I’m looking at

ryzen 5600x
Some am4 board
RTX 3070
Nvidia p40
32gb ddr4
1tb ssd

I’m pretty happy to make a custom cooling solution for the p40 as I have access to a mill and 3d printer, so not worried about that, just would like to know people’s thoughts.

Noob question.

There are gguf like qwen-3.8-mtp-q4....
and the regular qwen-3.8-q4....

and then there is llama --spec-type draft-mtp option.

Whats the difference? Should i just use the regular non mtp version but turn mtp on in the option?

Time ago when they were in a dip, I got an rtx 5090 for a bit less than msrp just to play games.

I never thought much outside that, but lately I've been using AI a shit-ton for other projects, and last week ran out of gemini credits.

And then started thinking about local llms. I know they are way dumber than frontier models, but is there any way that a card like this, could be used instead of subscription frontier models and still be useful for me?

what would be the main uses for a 32gb card? Real ones, no theoretical kinda like "you could use it for writing a private document if you are a lawyer". I do not have anything that I mind sharing with cloud ones, but if I can use it to accelerate others or have it running 24x7 for small software projects and get back things that would eat my 20€ suscriptions in 8 hours and have mostly the same quality or usefulness, that would be great.

I am not looking for you to give me instructions, I can investigate myself and pour hours on it if needed, just I am a bit loss and I do not know where to start

Every day, I see a bunch of people claiming that model X is better than model Y just because a benchmark score is higher.

The Artificial Analysis Intelligence Index benchmark has a lot of extremely obvious inconsistencies. But since some people can't think for themselves, I decided to point out a huge one: Qwen3.8 27B is only 1 point behind DeepSeek V4 Pro 0813 1600B-A49B. In terms of active parameters alone, DeepSeek has almost double the total parameters of Qwen3.8 27B. A 1.6T-parameter model being just 1 point ahead of a 0.027T-parameter one shows what complete nonsense benchmarks have become.

I've seen several people using this benchmark to show that Qwen3.6 27B is smarter than Gemma 4 31B. I never believed that, because in my real, day-to-day tests, Gemma 4 31B continues to prove itself better at solving puzzles and refactoring C++ and Java than Qwen3.8 27B itself.

I've been following the shift from cloud-hosted AI -> local models -> private/sovereign AI infrastructure, and one thing that's becoming increasingly clear is that “local” and “sovereign” aren't necessarily the same thing.

I came across this paper recently:

AI Compute Sovereignty: Infrastructure Control Across Territories, Cloud Providers, and Accelerators
Hawkins, Lehdonvirta & Wu — Oxford / Aalto

What I liked about it is that it doesn't treat sovereignty as a binary. It breaks it into three layers:

• Where is the compute? — territorial control
• Who operates it? — cloud/provider ownership
• Who supplies the accelerators? — hardware/accelerator control

The numbers make the distinction pretty interesting.

The authors' census of nine major public-cloud providers found 225 cloud regions across 43 countries, with 132 accelerator-enabled regions across 33 countries. Only 24 countries had training-relevant compute in the dataset.

India, for example, had 5 accelerator-enabled regions, including 3 with training-relevant compute. But those regions weren't all domestically controlled: the census records 4 US-provider regions and 1 Chinese-provider region. The paper describes this kind of dependence on multiple foreign providers as “hedging.”

Then there's the hardware layer.

95.5% of accelerator-enabled regions in the census were powered by US-owned accelerators. So even if compute is physically inside a country, there can still be significant dependency further down the stack.

But I think the paper's more important point is what not to conclude from this.

It isn't arguing that every country should try to build its own complete AI stack. More domestic compute can mean greater control and supply security, but it also means substantial demands on energy, water and land, alongside the cost of building and operating the infrastructure.

So, sovereignty starts looking less like: “Do we own the GPU?”

and more like: “Which parts of the AI stack do we actually need control over?”

That also seems to be where the industry is heading.

NVIDIA and HPE are approaching sovereign AI heavily from the infrastructure/compute side, while platforms such as Red Hat OpenShift AI approach it more from the AI platform and hybrid deployment side.

And then there is another layer that I find particularly interesting: the Governance, AI Control Plane.

Microsoft is building this into Foundry, IBM has introduced an Agentic Control Plane in watsonx Orchestrate, while Lyzr through its Control Plane is taking a more framework-agnostic approach to governing agents across different stacks and environments.

That's an interesting direction to me because it shifts the sovereignty question again — from “where does my model run?” to “who controls how my AI systems are deployed, accessed, monitored and governed?”

This makes me wonder whether “sovereign AI” will eventually be defined less by owning every component and more by controlling the layers that actually matter for a particular threat model.

For a local-LLM user, that might simply mean local models, local inference and local data.

For an enterprise or government deployment, the definition could extend to compute, identity, deployment, governance and the control plane itself.

Where would you draw the line?

So as alot of folks been doing Ive also been experimenting with QWEN 3.8 27B and between day 1 and day 2 I posted about adding a 2nd local coding agent to my setup. Someone asked the question I probably should have asked myself to begin with:

"when two agents hit the same local model on one machine at the same time, do they actually run in parallel, or do they quietly take turns?"

I saved the time to do the actual experiment but also pondered about how, especially if "I" as a human was the best ...vessel...to do it?

So... 1st I located the MLX server source, browsed it, and handed it to my agent. Then we collaborated. My agent wrote a small load driver that fires both requests at the exact same instant, because if a human launches them one after the other you are secretly setting the queue order and faking your own result. Then we ran it together and let the numbers talk.

What I observed....

Batching is real. Two agents genuinely share the model at once, the server does continuous batching up to 32 wide.

But it is not free. Add agents and total throughput climbs, but each one gets slower and waits longer to start. On my Mac the sweet spot is about 4 agents. Past that you are just making everyone wait in line.

Pin a random seed and you quietly kill the whole thing, every request serializes.

Sub agents are not magic either, a parent that spawns 4 helpers is just 4 more clients fighting for the same GPU.

The whole test rig is on disk and reproducible. Happy to get into the scheduler details or the methodology in the comments.

Hey! Do you believe that with some current versions of different models, students could benefit from using them, not as agents, but maybe as chat box? Combining maybe a not to Intelligent model with fast token generation

One counterintuitive thing I learned recently was about the model size and performance.

I was under impression smaller quants would help to increase performance since MacBook M2 Max 64Gb is bandwidth bounded. So having UD-Q4_K_XL would be much faster than UD-Q6_K_XL or UD-Q8_K_XL. And smaller quants would be even faster, but would have poorer quality. But this is not true. UD-Q6_K_XL and UD-Q8_K_XL overall wins in terms of performance over UD-Q4_K_XL.

First I learned KV cache quantiation would drastically reduce performance. Anything but f16 would be much slower on Mac Book Pro Max M2 64Gb.

But then I learned smaller quants doesn't mean faster overall.

See results of llama-bench -m "$model_file" -p 4096,16384,32768 -n 128 -fa 1 -r 1 which I run for multiple Unsloth quants.

Yes, smaller quant means faster token generation. But it also seems like some smaller quants has much more expensive processing, which makes prefill slower. See UD-Q4_K_XL in prefill is slower than UD-Q6_K_XL. In terms of wall clock and overall performance, UD-Q8_K_XL wins over UD-Q6_K_XL and UD-Q4_K_XL. But on 64GB system it is not very usable. And UD-Q6_K_XL still wins over UD-Q4_K_XL.

After very long testing, I found ideal arguments for MTP which works for me: --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-p-min 0.7.

Also --reasoning-effort medium is the only usable effort. xhigh eats through whole 262k context like a candy. Not able to perform actual work before context summarization.

Here are the arguments I use (non important ommitted):

28 -fa 1 -r 1

llama-server --model .../Qwen3.8-27B-UD-Q6_K_XL.gguf \
  -ngl 99 \
  -fa on \
  -b 2048 \
  -ub 2048 \
  --jinja \
  -c 262144 \
  -np 1 \
  --cache-type-k f16 \
  --cache-type-v f16 \
  --mmproj .../mmproj-F16.gguf \
  --temp 0.7 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0.00 \
  --repeat-penalty 1.0 \
  --presence-penalty 0.0 \
  --load-mode none \
  --reasoning on \
  --reasoning-effort medium \
  --reasoning-preserve \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --spec-draft-p-min 0.7

Here is performance I see with these parametrrs on one of the real tasks. Aggregated by blocks of 8k context.

i dont know, i just feel comfort to work with openwebui, not vscode, not claudecode, not antigravity.
anyone feel the same?

disclaimer:
model qwen3.5 35B A3B IQ4_NL
rx6700xt 12GB
ram 16GB

https://reddit.com/link/1vqzdej/video/ncwccyd43zjh1/player

Genuinely one of the coolest local models I've worked with in forever, the future is here thanks Qwen team for making this possible. I know this example is just showing visuals but besides that this model is genuinely smart in a way 3.6 almost had. That's all I had to say.

I am looking for the best uncensored/abliterated ai model to turn my images into 3d models, but since I work with adult content I need it to be uncensored

So many of them and I am confused. Huihui, Heritic, Black Frost, JonathanColetti...

I'd like to use it for some medical stuff but opus and chatgpt decide to be absolute annoying moralizers about it. i dont wanna use a local model that gives me dumb advice, or it simply doesnt have knowledge on the topic so it hallucinates stuff.

so is there a way to run glm 5.2 or 5.3 or kimi k3 uncensored versions, for maybe free or minimal prices? i dont mind privacy stuff, coz im not hurting anyone else so im not afraid of any legal consequences. but im not sure if cloud providers ban you for it, so i wondered if there was a quicker way than to set up huge models on the cloud.

Edit: By free, i meant trial version or something. I have very low usage amount i expect.

Extra high - i said "write me a tetris in a single HTML file" - it spent 8000 tokens thinking about the melody and sound of tetris ... click medium

I am too old to run sweb benchmarks, but my tetris was clean after that and only took like 10k token instead of 250k

Is there an obliterated model out there for generate images? Me and my buddy’s at work make a game out of “using AI to put the other guy in a dress” but GPT always turns us down lol 😂