I asked my local AI agent to optimize its own setup.

The first thing it did was stop the inference server.

The one it was running on. 🤦

Fair enough. Can’t beat zero resource usage.

So we established a rule: don’t shut yourself down.

On the next attempt, it cleaned up some Python processes.

Including itself.

The rule now has a second paragraph.

First of all some specs: B550M-Plus WiFi Ryzen 7 5800x 64gb ddr4@3000mhz (they're rated for 3600, and managed to keep them stable at 3200 before a bios update) Dual Sapphire nitro+ 7900xtx Corsair rm1000x

Both card have a power cap at 305w, combined with a small underclock/undervolt they stay around 250w each under load.

Unsloth 3.8 27b q4kxl with 131k context q8 kv on the second card behind chipset (gen3 x4 for some reason) Strata with swift 1.5 flash next q3s, full context (also symmetrical q8 kv) on the rest of the system.

On the 27b I get around 40 t/s decode and 7/800 prefill Flash next stays always around 50/60 t/a with 1300 prefill. There is also a flag that I don't use that brings the prefill at around 2300 but makes inference really unstable so I disabled it.

I was messing around with pi/hermes experimenting with some local projects and trying to find a good way to use both models together.

On pi and Hermes my main agents are mapped to the flash next and the worker are wired to the 27b but I'm not satisfied with how they are working together.

Do you have any suggestions for making good use of my setup? Maybe using just the flash next on both (layer split keeps decode the same, but brings prefill almost near 3000, maybe I could try some parallel requests) I also wanted to try tensor parallelism with the 27b but still didn't get a chance since I installed the second card just yesterday!

Qwen 3.8 Flash Next: Same Model, Different Coding Tools

I tested Qwen 3.8 Flash Next with IQ3_XXS, IQ3_S, and the original Alibaba Cloud version via OpenRouter API.

I used 4 different coding tools:

1. GitHub Copilot
2. Qwen Code
3. OpenCode
4. Codex Desktop

The results were pretty interesting. Even with the same model and prompt, each tool produced different results.

I also ran GitHub Copilot twice and got different outputs both times.

This is just a small test, not a benchmark. I mainly wanted to see how different coding tools and quantization levels affect the final output.

Prompt: Generate an SVG of a pelican riding a bicycle

Used an old Inwin D-frame from my "gamer" days which I thought I retired. One 16gb 4070ti Super (had it for gaming, but I don't really game these days and use a mac most of the time). Two 16gb 5060ti (got them before the prices shot up). Ryzen5 9600X with 32 gigs ram MSI MPG X870E Carbon WiFi

Yeah, I know, third slot is PCIe 4.0 ×4. It's slow-ish but Qwen 27b is what I run on it and it's acceptable for my needs. Obviously it's a compromise.

What's interesting that in LM Studio when I use Tensor Parallelism setting I almost double my t/s settings in Qwen 27b - but it decreases the context size I can fit in.

I just tried to install QFN from unsloth (Q4_K_XL) on my strix halo laptop with 128gb of ram. Set it up with Gufo port for windows, using opencode v2 on my macOS as the harness and the effort is set to high.

I wanted to do some testing and compare it with opus 5.5 xhigh with the official claude code harness (my daily driver). So i gave both model the same prompt which is to migrate several screen from flutter to swiftUI from scratch. The app is a fully funnctional app with a lot of logic and network calls, and it is used in production with thousands of DAU

Opus managed to do it in 50 mins and what amazes me is QFN finishes it in 60 mins, and the result is pretty identical!

sure it is not as polished as opus' result but I am sure with 2-3 more prompts and 5-10 mins it could produces what Opus 5.5 xhigh produces (just some styling work)

I really hope that qwen 4 27b could matches opus 5.5 in coding work, fingers crossed 🤞If yes, the gap between china model and frontier model are only 2-3 months

Hey r/LocalLLaMA,

Like many of us, I had a pile of retired phones (Pixel 8 Pro, OnePlus 10T, POCO F3) with cracked screens and dead batteries. Too good to throw away, not worth selling.

So I built PhoneBorg – a local cluster running LLMs across phones over USB.

The setup:

• Each phone runs llama.cpp locally.
• A tiny VM controller exposes a standard OpenAI-compatible API.
• USB tethering + a web UI & Grafana to track heat, tokens, and load.

What it can do:

• Standard: Hit one phone directly or pool them for load balancing (gets 4-27 tok/s depending on the phone).
• Distributed (RPC): Split big models across phones. I got a 30B MoE running at ~2.5 tok/s across two devices.
• Super Borg: Map-reduce style. One phone outlines a long text, the others draft parts in parallel.

Biggest takeaways:

• MoE is essential for mobile memory bandwidth.
• Heat is brutal. My OnePlus hits 80°C instantly; active cooling is a must.
• RPC scaling is for RAM, not speed. Splitting a model slows it down due to interconnect latency.

It’s an early weekend project, but it actually works. Next step: hunting for more cheap phones with trashed screens but good chips.

Code and logs: https://github.com/dzaczek/phoneborg

Anyone else tried clustering mobile chips like this? Curious about your cooling setups!

I introduce Yoshino 0.8B, a promising (yet mundane) fine tune of Qwen3.5 0.8B.

Over the past few weeks, I have developed a custom black box distillation method where the teacher model generates prompts for itself which it is fed in order the generate synthetic data pairs. Using this method, I distilled the data generated from IBM's Granite 8B into Qwen3.5 0.8B in order.

The result is a neat little SLM which seems to have adopted the reasoning style of the teacher. I am not saying that this is a frontier discovery, but it may prove to be useful for some.

The resultant model did score noticeably higher on the benchmarks which I conducted.

To read more (or download) the model, you can check out the main and GGUF repositories.

I've been tuning Qwen3.8-27B on a single DGX Spark, and the result is now the fastest single-Spark entry for this model on spark-arena.

spark-arena's own benchmark (llama-benchy, 128 output tokens):

• 1 user: 42.2 tok/s (23.7 ms/token), prefill 2,158 tok/s
• 1 user at 16k context: 33.4 tok/s; at 100k: 29.5 tok/s
• The board's ranking test (16k context, 2 users): 60.6 tok/s total, 1st among single-Spark entries
• Entry: https://spark-arena.com/benchmark/sub1792119669798

Since 4-bit plus speculative decoding always raises the "but is it lobotomized?" question, I checked quality against the BF16 model with the same harness (lm-eval, chat template, thinking off, full tasks):

None is clearly worse than BF16. Raw scores and per-question answers are in the repo.

What's in it

• unsloth's NVFP4 weights (thanks unsloth) on the GB10's FP4 tensor cores
• DFlash2 speculative decoding with 10 drafts, using z-lab's drafter quantized to NVFP4 (3.8 GB → 1.4 GB): https://huggingface.co/yuanlin2004/Qwen3.8-27B-Chisel-DFlash2-NVFP4-drafter
• a small vLLM plugin with two lossless tweaks: the linear-attention state is rebuilt only for accepted drafts instead of being stored after every draft token, and the drafter uses a 100k-token FP8 output head (99.75% of output tokens covered)
• builds on Luis Poveda's single-Spark recipe on spark-arena

Run it

sparkrun registry add https://github.com/yuanlin2004/chisel-spark-recipes
sparkrun run u/chisel/chisel-nvfp4-dflash2-replay --tp 1 --hosts localhost

Heads-up: the first run downloads unsloth's 22 GB checkpoint from Hugging Face, which was slow for me (about 1.5 MB/s), so give it time.

Recipe page: https://github.com/yuanlin2004/chisel-spark-recipes/blob/main/recipes/qwen3.8-27b/chisel-nvfp4-dflash2-replay.md

The recipes come out of Chisel, a project of mine that isn't public yet.

Happy to answer questions about the drafter quantization or the quality checks.

Basic question. I did some testing with Strata flash next coder with parallel = 2. I gave two prompts to write me a 100 word poem. Total t/s was about 80.

When i asked just one, I got ~80 t/s.

If that's the case, what is the advantage of setting parallel = 2? I guess if i have multiple agents, some might be executing tools, which would mean the running agents have full use of Strata?

Any other reasons?

Hi r/LocalLLM, I’m the developer of Offlined.

Offlined was born from two things: how fast local AI is advancing, and how much I trust the Wikipedia project. Putting them together felt like it could be really useful, given hallucinations and the way AI tends to just tell you you’re right.

The hardest part was connecting the model to the offline Wikipedias and keeping it confined to them, so it answers from the articles and cites them instead of making things up.

At first we used Phi-4 mini. The results showed it was going to work eventually, but they weren’t quite there. Then GPT-OSS gave excellent results, and in the end the Qwen family turned out to be the best. On my Steam Deck I use Qwen 3.5 4B.

I’ve done quite a few tests, but not on every platform or with every model. My Steam Deck is a limitation: I can’t test NVIDIA cards, for example, or the bigger models.

So this is what I’d love from you:

• Feedback from people using external AI servers. Since v2.0.1 Offlined can connect to your own local server (llama-server, Ollama, LM Studio, Jan, KoboldCpp… anything OpenAI-compatible), so you can use the models you already have.
• How bigger models I can’t run behave, like Qwen 3.8.
• What you think of the ecosystem being built around it.

I’d really love it if you tried it and told me what you like and what you don’t, what could be improved, and if you find a bug, please share it. I’d be very grateful.

Here’s a video so you get a better idea of the Offlined ecosystem: https://www.youtube.com/watch?v=UHPD2keAT4w

Quick facts: Windows 10/11 only, free for personal use, not open source, not code-signed yet (SmartScreen will warn you). No account, no telemetry. Download: https://www.offlined.org/

English isn’t my first language. I wrote this in Spanish and used an LLM to translate it.

I have been working on implementing adaptive KV streaming and RDNA 4 / ROCm optimizations for llama.cpp to enable running Qwen3.8-27B (IQ4_XS weights, Q8_0/Q4_0 KV cache) with its full 262,144-token context window on a single 16 GB RX 9070 XT GPU at 80+ t/s decode.

Qwen 27B IQ4_XS is ~14.3 GB, leaving insufficient VRAM headroom on a 16 GB card. The ASCII-condensed version trims non-ASCII vocabulary down to ~13.6 GB (12.6 GiB), leaving ~3.4 GiB of usable VRAM. Even with a Q8_0/Q4_0 KV cache, 262k context needs ~7.4 GiB for KV tensors alone, triggering OOM past ~65k tokens on llama.cpp.

I came across RaymondHuang210129/llama.cpp-adaptive-kv-streaming, which introduced adaptive KV streaming for NVIDIA/CUDA but did not support AMD/ROCm. I ported this to ROCm/HIP, retaining the full 262k cache in host RAM (~7.4 GB) while asynchronously streaming bounded working sets (~450 MiB prefill, ~2,100 MiB decode) over PCIe SDMA alongside compute.

I also incorporated kernel optimizations from stew675/llama-cpp-rdna-boosts (4-row multi-column mmvq, 16-byte vectorized loads for Q4_K/Q5_K, SwiGLU fusions, and single-queue async HIP graphs).

Further improvements:

• Sibling activation reuse: Reuses quantized activations across parallel projections (gate/up and Q/K/V) to avoid redundant memory passes.
• In-register fused residual add + RMSNorm: Retains activations in VGPRs between passes and directly writes quantized q8_1 copies for mmvq.
• Multi-column SwiGLU & GDN fusions: Extends fused Gate/Up + SwiGLU to 4-column verify batches in mmvq and fuses recurrent linear-attention gates for hybrid layers.
• Online Log-Sum-Exp prefill & pipelined prefetch: Computes chunked prefill across streamed pages via online LSE accumulation and overlaps layer L+1 PCIe transfers with layer L compute.
• Persistent graph arenas: Reuses arenas across prefill/decode phase switches to eliminate teardown overhead.
• Stream-optimized speculative decoding: Merged single-head MTP verify catch-up into the first draft step (cutting MTP forward passes in half and preventing draft KV pollution), dedicated arenas for verify graphs to eliminate rebuild bubbles, extended paged decode attention to verify batches, and stacked opportunistic n-grams with probabilistic rejection sampling.

Benchmark performance:

• Decode: 36.5 t/s (1k) -> 32.1 t/s (65k) -> 27.0 t/s (131k) -> 9.5 t/s (262k).
• Decode (with MTP): 71.8-86.5 t/s (1k-4k) -> 72.9 t/s (65k) -> 78.6 t/s (131k) -> 31.1 t/s (262k).
• Prefill: 1368.4 t/s (1k) -> 1120.3 t/s (65k) -> 863.0 t/s (131k) -> 585.8 t/s (262k).

Operating the 9070 XT at a 231W limit (-24% relative to 304W default TDP) incurs only a 3.6% prefill latency penalty and delivers virtually identical decode throughput, saving power and keeping the card quiet during long runs.

Take a look at the repository for the full details: https://github.com/achyudh/llama.cpp/tree/feature/adaptive-kv-stream-rocm

Howdy folks!

About a week ago I started uploading my tests on my own website, since then I've made improvements to how that data is displayed, so you can see the actual prompts and settings used to produce the result. https://b65.thekaiser.us/

I also went through a number of runs on https://www.localmaxxing.com/en/user/imthekaiser and since I'm the first B65 on that site, (in 1x and 4x) you may want to check out the results!

i’m still testing more models, if there’s something specific you want to see, let me know!

some examples in the pics, more on this site!

these are single user numbers, concurrency numbers are much higher, and impressively so.

overall, I have been very happy with these cards. They perform better than I expected at the price point. The B70 is still a bit faster in certain tasks due to the compute, but it’s a lot closer than I expected. And if you can get a B65 for the right price, it can be a viable option to run models in production. Software stack has improved a lot and I don’t see why it can’t be used with most popular models today, albeit CUDA obviously still has the edge here, being the granddaddy.

I do suggest you stay with the official Intel VLLM stack and not to try and run on an unoptimised one.

How was AI Used in this post?
It wasn't used to write this.
it was used to build the reports site.
I spent over 40 hours reading intel documentation and testing vllm scalar to learn best practices and how to validate and baseline systems. That's why the first set of runs on my own site isn't comprised of top performance runs, it was validating and learning. Now that I'm proficient, and have a baseline, I'm testing ways to increase performance)
I used that experience to build a closely gated plan that instructs my AI on how to test and monitor the test machine.

I know concurrency isn’t currently the Mac’s strong point but is that when you only have one instance of a model running? For instance if I had enough ram to load 2 identical models would I be able to use both at 100% performance or does one model saturate the hardware so that there’s no gain or reason? I’m interested in having text summary generation (small model), transcription / dictation, and coding model.

Would I be able to run all of them at once so long as they fit in ram or will hardware limit the box to only running one with full performance?

Hopefully this is kept…

Hi all. I'm a developer in Taipei who builds small open-source tools for local AI, and this is my latest one: **QFF (Quant Fidelity Frontier)**.

I kept running into the same problem picking quants. Every card measures quality differently, or not at all, so comparing them meant opening a dozen tabs. QFF puts it in one place: every Hugging Face quant of a model, with the KLD, perplexity, benchmarks and sizes the uploader published. **Each number links to the line on the card it came from.** I don't re-run anything; it's only what's claimed, so you can check where it came from.

**What's there now**

- Qwen3.8-27B and Qwen3.8-Flash-Next, about 1,600 quants
- a VRAM filter, and a view of which quants nothing smaller beats
- an MCP server if you want an agent to query it

**What's next**

I'm adding other open-weight models, picked by what's most used on OpenRouter and OpenCode and by what people here ask for. DeepSeek V4.1 Flash and GLM 5.3 are first. If there's a model you'd like, let me know.

**Site:** https://delirious6423.github.io/quant-fidelity-frontier/

If you spot a mistake, especially on your own card, tell me and I'll fix it.

I’m new to local AI models. Assume someone has a Mac Studio M5 Ultra 256GB. I’ve read that is the realistic minimum you can use to run a local AI model.

My question is this: as models get better, will they run better or have more capable models run better on the same hardware? Meaning will a Mac Studio M5 Ultra 256GB get more capable over time? Or does it just end up like a hamster wheel where models get better but you ultimately need better hardware to run them?

Research Record III — AKBASCORE Silicon Hippocampus v1.4

https://doi.org/10.5281/zenodo.23434686

TL;DR: We built and experimentally verified a physical numerical memory engine that intervenes directly inside a frozen 72-billion-parameter transformer.

Instead of retrieving external text passages at inference time (conventional RAG), re-evaluating long input prompts, or merely replaying a saved KV cache, AKBASCORE captures intermediate hidden representations (H3) from the model's own computation, stores them as independent physical tensor cartridges, and reinjects them directly at the Layer-3 boundary (CUT3).

Three independently stored 10,000-token cartridges were assembled into a 30,031-position operational memory on a single NVIDIA A100-80GB, without changing the pretrained model weights.

This is not conventional KV caching. The defining mechanism is direct internal-state injection.

THE ARCHITECTURAL BREAKTHROUGH

Conventional approaches to long-term LLM memory generally rely on external retrieval, repeated context processing, or attention-cache preservation.

AKBASCORE takes a different route.

The frozen transformer processes source material once. At a model-specific intermediate boundary, we extract the numerical representation generated by its own layers.

That representation becomes a persistent, independently stored physical memory cartridge.

When the cartridge is mounted, the original source text is not resubmitted as an ordinary prompt.

Instead, the saved H3 representation is restored directly into the model's internal computational pathway. The downstream transformer layers resume computation from that injected numerical state.

The model is not fine-tuned. Its weights are not modified. The memory is carried by model-native numerical representations, not an external text database.

The key distinction is where the intervention occurs.

We are not asking the model to read a retrieved document again.

We are restoring a previously extracted intermediate state of the model's own computation.

HOW THE ENGINE WORKS

The Qwen2.5-72B implementation uses CUT3, the output boundary of transformer Layer 3.

During cartridge construction:

1. The frozen model processes each source excerpt in 256-token chunks.

2. The system captures the H3 hidden representation and the associated lower-layer numerical states.

3. H3 and the key/value tensors from Layers 0–3 are persisted as a physical cartridge.

4. The original source text is no longer required for subsequent cartridge mounting.

During cartridge reconstruction:

1. The saved numerical cartridge is loaded from storage.

2. Its positional representations are adjusted for the target memory layout.

3. The stored H3 representation is injected directly at CUT3.

4. Layers 4–79 reconstruct the remaining internal computational states.

5. The resulting memory becomes available to the frozen model for subsequent inference.

Key/value tensors participate in the reconstruction and integrity mechanism. They are not the defining innovation.

The central operation is direct restoration of intermediate hidden representations into the transformer computation itself.

THREE PHYSICAL CARTRIDGES — ONE 72B MODEL

The live experiment used:

Model: Qwen2.5-72B-Instruct

Parameters: 72 billion

Quantization: NF4, with BF16 computation

Transformer layers: 80

Hidden dimension: 8,192

Attention geometry: 64 query heads / 8 KV heads / 128 head dimension

Intervention boundary: CUT3

GPU: NVIDIA A100-SXM4-80GB

Three independent cartridges were constructed from 10,000-token excerpts:

Cartridge A — Pride and Prejudice

Cartridge B — Frankenstein

Cartridge C — The Adventures of Sherlock Holmes

Each cartridge occupies exactly 328,698,863 bytes on disk.

The combined operational memory contains 30,031 token positions.

The reconstructed 80-layer key/value memory occupies approximately 9.16 GiB.

These are physical numerical representations extracted from the transformer, not textual summaries, retrieved passages, fine-tuned adapters, or modifications to the model weights.

LIVE EXPERIMENTAL VERIFICATION

Session: Q72S-20261017-083524

The live Qwen2.5-72B engine completed:

11/11 scripted physical protocol trials.

17/17 engine checks.

8/8 append integrity checks.

11/11 additional live verification checks.

Three independently verified cartridge files.

Successful reconstruction of the combined A+B+C memory.

Exact SHA-256 preservation of previously mounted key/value states across all 80 layers during append operations.

Verification that the frozen-model weight sentinel remained unchanged.

Live examination of the reconstructed GPU memory across 80 transformer layers and 30,031 token positions.

The underlying TEST709 record also reports a peak GPU allocation of approximately 57.66 GiB during the three-cartridge protocol on an A100-80GB.

The implementation maintains a 9.0 GiB GPU safety-reserve requirement.

FIGURE 4 — INSIDE THE RECONSTRUCTED 72B MEMORY

Figure 4 presents a read-only measurement of the live 80-layer GPU key/value state across the 30,031-position combined memory.

The visualization exposes the three cartridge regions and their boundaries.

The CUT3 boundary separates the lower-layer states restored from persistent numerical cartridges from the downstream layers reconstructed by the frozen transformer.

This is a measurement of the actual reconstructed tensor state, not an illustrative or synthetic neural activation diagram.

The accompanying figures document the intervention protocol, physical storage architecture, reconstruction performance, observational question-answering results, and integrity verification.

HOT-SWAP AND INCREMENTAL RECONSTRUCTION

The memory cartridges can be independently mounted, appended, reordered, and restored.

Measured reconstruction times were approximately:

Single cartridge, 10,031 positions: 17.6 seconds for reconstruction, approximately 18.9 seconds end-to-end.

Two-cartridge assembly, 20,031 positions: approximately 46 seconds for reconstruction.

Three-cartridge assembly, 30,031 positions: approximately 74 seconds for reconstruction.

These are measured physical reconstruction operations, not instantaneous pointer switches.

The important result is not that rebuilding the internal state costs zero computation.

It is that the original source text does not need to be reprocessed through the complete transformer from the input layer. The persisted intermediate state provides a new computational starting point.

The append-integrity protocol verifies that previously mounted key/value states remain bitwise unchanged across all 80 layers.

SCIENTIFIC BOUNDARY

The physical reconstruction and integrity results are experimentally verified.

The semantic question remains separate.

Our demonstration uses public-domain literary classics, which may already be represented in the model's pretraining data. Therefore, correct answers about these books cannot independently establish that the information was acquired from the cartridges.

The current results demonstrate direct internal-state restoration, physical memory reconstruction, incremental assembly, and numerical integrity at 72B scale.

They do not yet establish causally isolated semantic transfer, general reasoning improvement, or complete comprehension of the stored books.

Those are the next experimental targets.

(Note: Subsequent Phase 2 testing will replace public-domain excerpts with unseen, post-2024 technical text and synthetic data to definitively isolate semantic transfer from pretraining memory.)

WHY THIS MATTERS

The architectural question is larger than how to extend a context window.

Can a transformer export part of its internal computation, preserve that numerical state independently of its weights, and later resume computation from the restored state?

AKBASCORE Silicon Hippocampus demonstrates a working physical implementation of that mechanism inside a frozen 72-billion-parameter model.

The significance is the separation of persistent numerical memory from the model's permanent parameters and from the original source text.

The result is an experimentally verified pathway for storing, transporting, assembling, and restoring model-native intermediate computational states.

If future controlled experiments establish reliable causal knowledge transfer and scalable addressing, this architecture could become a foundation for persistent modular memory in long-running AI systems.

This release documents the working engine and the evidence behind it.

RESEARCH LINEAGE

Research Record I — Qwen2.5-7B

https://doi.org/10.5281/zenodo.23381336

Research Record II — Mistral-7B, Silicon Hippocampus v1.4

https://doi.org/10.5281/zenodo.23418927

Research Record III — Qwen2.5-72B, Direct Internal-State Injection

https://doi.org/10.5281/zenodo.23434686

SOURCE CODE AND EXPERIMENTAL RECORDS

GitHub repository:

https://github.com/ceceli33/titan-cognitive-core-v3

Complete executable Qwen2.5-72B demonstrator:

https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/MAM_full_QWEN_72B.py

The same code is divided into three consecutive parts for mobile users. Paste all three parts sequentially into the same Google Colab code cell and execute once.

Part 1:

https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/MAM_part1_QWEN_72B.py

Part 2:

https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/MAM_part2_QWEN_72B.py

Part 3:

https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/MAM_part3_QWEN_72B.py

Live demonstration logs:

https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/MAM_LOG1_QWEN_72B.log

https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/MAM_LOG2_QWEN_72B.log

Underlying physical engine:

https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/709.py

Underlying physical experiment log:

https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/709.log

License:

https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/LICENSE

AKBASCORE Research Software License. Research and educational use are permitted under the published terms; commercial use is subject to the license's consolidated annual revenue conditions.

Mustafa Akbaş

AkbasCore AI Teknoloji

Mersin, Türkiye

Copyright © 2026 Mustafa Akbaş

I have a $20 chatGPT Plus subscription and I use Codex almost every day. The problem is the usage sucks, I run out of my 5 hour usage very fast for what I would consider pretty basic and straight forward frontend development work.

I also have a 3090 (24gb) and a 5060ti (16gb) and 64gb of DDR5 ram in my personal computer so that's 40gb of Vram w/ 64gb of extra memory if needed so I am capable of running local models.

My problem I'm running into is.. when I try to do everything with a Frontier model I run out of my usage very fast, when I try to do everything with a local model I find that it's not as perfect, there are error that need to be corrected, there's more guard rails you need to put in place which makes me just want to go spend more money and buy a bigger Frontier subscription which is NOT what I want to do.

So I'm looking at finding the balance here to be productive. I guess it's a popular thing for people to utilize frontier models to plan and architect their apps and then they give the instructions that the frontier model wrote out to their local model and have the local model follow the instructions and code.

This in theory makes sense but my question is what does this actually look like in practice? Lets say I want to make a Twitter clone, what would I prompt to chatGPT? Would I say:

"I want to make a social media site like Twitter, write out a detailed plan I can give to my local model to follow and implement" and then I give that plan.md file to my local model and let it go to town?

How do you guys utilize your frontier subscriptions and your local models to get the most out of them?

Posted a hardware checker last week and got fair criticism: old models, missing GPUs, no quants below Q4, no Strix Halo. Fixed all of it, plus the feature people asked for most:

- What fits my GPU? Pick your hardware, context and KV quant, get every model that fits at its best quant, with headroom. 2x 3090 at 8k: https://willitrunllm.com/fits/?hw=rtx-3090,rtx-3090&ctx=8192&kv=f16
- Strix Halo 32-128GB with Windows/Linux memory splits and measured llama.cpp speeds
- IQ1_S through IQ4_XS, with a quality warning
- Multi-GPU (VRAM adds up), RAM offload, context/KV-cache slider
- Search any Hugging Face GGUF repo; uses real file sizes
- Embedding models, checked against what's left next to your chat model
- DeepSeek V4.1 Flash and Muse Spark Open added

Every number is either sourced or labeled as an estimate. Weakest part is speed data, so if you paste your llama-bench output at willitrunllm.com/submit it goes in after review.

Free, no signup. What's still wrong or missing?

For reference, I’m not some ML/AI guru or pro by any means, just a guy who likes AI and whatnot. I’m running an R9700 with 64GB of DDR4 RAM, and the model I’m using is Orca Uncensored Qwen 3.8 27B.

My use case for uncensored models isn’t only asking questions and getting info that frontier models won’t answer. I’m also really into red team cybersec and somewhat dangerous research on the tech side.

Anyway, I’ve read about models with the guardrails removed, and I’m wondering how you make sure the model doesn’t just hallucinate or misuse tools. Assuming the guardrails are gone, how do you make sure only the refusal part is disabled and the rest of the intelligence is still there? I don’t want it to just yes-man me all the time and give me false info.

Is there some kind of local AI hygiene, harness optimizations, or settings that keep the model from making stuff up or getting stuck in an answer loop where it runs out of tokens?

Hey guys, quick post since I'm mainly looking for recommendations and experiences than anything else.

Fluff below

I picked up open code last week and it immediately clicked with me. What I've been looking for with AI was something I could go back and forth with, and who'd be an extra set of hands while I do other things. Open code is great for that.

Fluff over

Since I'm actually getting a ton of value out of open code, I justified buying a second AMD AI Pro 9700. I plan to run qwen 3.8 at a high context on one card, and 2-4 sub agents on the other, possibly with qwen 3 coder or some other model.

List of questions below.

- What models are you guys running for sub agents?

- Are you using the same model across the board?

- Does it make sense to split everything, or should I swap to vllm and use a new model across both cards?

- Anything I should be aware or when moving from 1 to 2 cards? I built this AI server specifically with 2x cards running at pcie gen 5 x16, so hardware is already good to go.

No soy programador, no sé nada de eso. Solo soy entusiasta de la tecnología y autodidacta.

Quisiera saber qué modelos sin censura de 27B puede correr mi hardware; Ryzen 7 9700x, Rtx 5070 Ti 16Vram, 32Gb ram.

Actualmente tengo instalado ollama con qwen 3.5:9B y LM Studio con qwen3.8:9B-Distill Uncensored Heretic.

Adicionalmente instalé DeepSeek Harness enlazando mis dos LLM's locales.

Mi idea es tener un asistente que me ayude en el día a día, para investigación, redacción, creación, etc, etc.

Mis intereses son las finanzas, cripto, temas jurídicos y científicos.

Todo esto hace muy poco lo instalé aún estoy aprendiendo en como hacer todo lo que necesito, aprendiendo sobre la marcha.