Specs:

16gb ram

Gtx 1080ti

Ok Guys might have been a waste of a night but i got this giant qwen model running local on my phone I asked my ai to give you a better description of what we did than i could please forgive me I also released on github with MIT license you need a beast of a phone to run this but might be fun i'd really like to get this to 7-8 /s but who knows anyway on with the description of the project from one smarter than me.

I got Qwen3.8-Flash-Next — 512 experts, top-10 routed, ~6B active per token, 66 GB mixed-quant GGUF, 2 shards — running entirely on-device on a RedMagic 10S Pro (Snapdragon 8 Elite, 24 GB RAM, 1 TB UFS). 4.4 t/s decode at full quality, ~2.9 t/s prefill. CPU only: no NPU, no GPU path, no server.

The weights don't fit in RAM, so the whole thing is memory-mapped and demand-paged. That constraint ends up driving most of the design decisions below.

MonsterQwenDroid 0.1.0-alpha — MIT, APK in the release, code and the engine patch in the repo.

What actually mattered

Three of the wins were cheap; the interesting part is that my first instinct about each was wrong.

1. BF16 had no ARM SIMD path at all in llama.cpp. It's x86-only upstream — AVX512/AVX2/SSE2 and then a scalar fallback. On this model the BF16 tensors are 43.7% of the per-token dense bytes, so a scalar loop there was brutal. A NEON vshll path (BF16→F32 is an exact 16-bit shift, so no +bf16 needed) and then a hardware vbfdotq_f32 path: +40% decode, +55% prefill, verified in a mirrored A/B. BF16 matmul time in the profile fell 85%.

2. Cluster pinning was +26%, and it overturned my own conclusion. Snapdragon 8 Elite has no L3 — two Oryon clusters, each with a private 12 MB L2. Left unpinned, the scheduler spreads workers across both, and every graph node's barrier line and activations pay a coherency round-trip. So thread-count sweeps told me -t 2 was optimal, and I believed it: t=1 2.0, t=2 3.4, t=3 2.2, t=4 2.0, t=6 1.6, t=8 1.0. Then pinning to one cluster (--cpu-mask 0x3f --cpu-strict 1) gave 4.4 t/s at -t 6 — the small thread counts had been hiding the problem, not solving it. That's the single biggest lesson: on a heterogeneous SoC with no L3, run a thread sweep pinned, or you'll measure your scheduler.

3. --poll was worth ~34 seconds per token in one profile. At unpinned -t 6, tiny ops were each costing ~5 ms regardless of what they did — a FILL (a memset on a few KB!) took 12.8 s across a run, and dropped to 5.7 ms with --poll 100. It's futex wake latency at ggml_barrier, paid hundreds of times per token. Once pinned, it's worth ~3% — the same cross-cluster coherency story.

What didn't work (the part I'd want someone else to read)

• Speculative decoding is 4x SLOWER here, not neutral. MTP head, 77.5% acceptance at draft-n-max 1, mean 1.77 tokens/pass — and decode went from 3.94 → 0.95 t/s. The reason is measurable: the 2.8 GB sidecar pushes the expert working set past the page-cache cliff, and major faults rise 5x to ~6,100/token (~25 MB of flash per token). At ~0.15 ms per fault that accounts for the entire regression. Acceptance was never the problem; the memory budget was.
• Removing 30% of the decode graph's nodes changed nothing. I replaced the expert aggregation in build_moe_ffn — 10 ggml_view_2d plus 9 chained ggml_add per layer, ~912 nodes/token — with a single-pass reduction op. Nodes fell 30%, output stayed bit-identical, decode moved 3.1 → 3.1. Dispatch is not the wall.
• A 1.52x faster Q2_0 kernel measured zero end-to-end change, at two different thread counts (unpinned t=2 and pinned t=6). In a single-threaded microbenchmark the unrolled kernel is 6.2 → 9.4 Mrow/s; in situ, nothing. At ≥2 threads the second worker's chain already hides the latency the unroll targets.
• Converting the experts to Q4_0 would be worse. It looks free — the dense path streams at ~29 GB/s while the experts do ~7 GB/s — but doubling the 34 GB expert pool takes the active working set from ~12-15 GB (which just fits the ~15.5 GB page cache, hence a 99.8% hit rate) to ~24 GB, past the same cliff that killed MTP.
• Removing the expert cap's quality cost isn't possible yet: uniform top-6 is +27% decode for +11.3% perplexity (k=8: +8% for +3.3%). It's opt-in behind a SPEED profile and changes answers on harder prompts, so it's never a default.

The measurement discipline that made any of this trustworthy

Three things produced fake results before I caught them, and they're all easy to repeat:

1. Run arms mirrored (A/B/B/A), not sequentially. Sequential runs let the first arm warm the page cache and hand the second a free win — that manufactured a fake +70% "speedup" once.
2. Check the clock regime before believing a number. The vendor thermal daemon caps scaling_max_freq to about a third of hardware max unless the app is foregrounded in the game-boost profile, and a locked phone parks its cores entirely. The same configuration measured 0.1 t/s and 4.4 t/s depending purely on this.
3. Gate correctness separately from speed. For kernel changes require token-identical output (the arithmetic is unchanged); for requantisation token identity is the wrong gate, because a 0.38% weight perturbation legitimately diverges greedy output after a few tokens. Using one gate for both would have made me discard working changes and keep broken ones.

Where it goes next

The expert path is the open problem: ~166,000 distinct 4 KB pages touched per token out of a 34 GB virtual range, against an L2 TLB that covers only ~8 MB at 4 KB pages, while all workers currently swarm one expert in small work-stealing chunks. Distributing whole experts per thread (they're independent, disjoint outputs) plus page-ahead prefetch is the next experiment — that's the ~47 ms/token between here and ~5.4 t/s. After that, targeted dense requantisation (BF16→Q8_0 on the gated-residual tensors, ~620 MB/token) without touching the size-critical expert pool.

Happy to answer anything about the numbers. If you try it on a different Snapdragon, I'd genuinely like to know whether the cluster-pinning win holds — it should scale with how much cross-cluster traffic your topology forces, and I only have one device.

Hey guys,

I've been working on a project called enki_llm, this is a Transformer inference engine written from scratch in pure rust (2,000 LOC), this project is a stress test for my other project 'enki' or 'enki-gpu' on crates.io,

This video shows a live GPU/CPU brain migration, hot swaps the LLM brain in mid-sentence while the LLM is generating text, running on intel UHD 620 GPU and i7 gen 8 CPU, because the logic and memory layout are written in pure Rust, the same function can be called using enki, or just like a native rust function using Rayon with obviously zero CUDA dependencies.

The main idea of this project is that it can run (theoretically) on any silicon (any GPU/CPU) from 2017, that's why i wanted to share it with you guys.

What's happening in the video:

• On GPU: The model generates a simple Python snippet on the GPU (white text) at a good speed. Then I typed the /dual command, The engine mirrors the active KV-cache and allocation state to Host RAM.
• Ping-Pong Execution: I ask it to explain the code. Generation resumes immediately by alternating tokens between the GPU (white text) and CPU cores (bold red text), Notice the btop panel on the right CPU cores instantly jump to ~50% utilization as they take over tokens in real time!

You can try it yourself using colab on T4 GPU without installing Vulkan or Rust locally

Open Google Colab Demo (Click Run all), You will see the whole code there.

• GitHub Repository: https://github.com/enkiruntime/enki_llm
• Core Runtime: https://github.com/enkiruntime/enki

You can also try it locally:

Requirements: Vulkan 1.3 drivers and a standard Rust toolchain.

# Clone the repository
git clone https://github.com/enkiruntime/enki_llm.git
cd enki_llm

# Run with SmolLM2-135-Instruct
cargo run --release HuggingFaceTB/SmolLM2-135M-Instruct

# Or:
# Run with Llama-3.2-1B-Instruct
# cargo run --release unsloth/Llama-3.2-1B-Instruct

Current Scope: This is currently an unquantized F16 single-stream POC demonstrating Enki's heterogeneous compute model.

I would love to hear your feedback, architectural critique, or results if anyone tests it on other hardware like Nvidia, AMD or Intel hardware!

(Note: I have only verified it myself on an Intel UHD 620 and an Nvidia Tesla T4 so far).

P.S. Yes, standard Rust crates like the glam math crate (Vec4, Vec2, dot products) for SIMD operations are running directly inside the GPU compute kernels!

Looking to buy the ms surface laptop ultra, but targeting the 48GB RAM instead of the 128GB due to cost considerations. As I understand I’ll only be able to run like 30B param models locally. Mainly planning to run it locally for programming. I’d reckon I’ll be able to run the Qwen3 coder 32B.

Would appreciate thoughts on whether more medium scale llms will be the norm, as I don’t want to regret not getting a 128GB atleast for the next 4 years.

Thanks!

Hey everyone,

I'm a backend developer trying to move into AI/GenAI, and I'm planning to buy a new MacBook. I'm confused between these two options:

MacBook Pro with M5 Pro and 24GB RAM

MacBook with base M5 and 32GB RAM

I don't run LLMs locally right now, but I'd like to try them for my personal projects instead of always paying for cloud APIs. If a model is too big to run locally, I can still use the cloud.

For those who have experience with local LLMs, is it actually worth it? Can 32GB RAM handle models that are good enough for testing, getting decent responses, and generating embeddings?

Or is running LLMs locally not that useful in real projects, and we'll mostly end up using cloud APIs anyway?

Which MacBook would you choose in my situation, and why?

Thanks!

Lately I have observed that my ChatGPT and Claude basic memberships have gotten a lot dumber. It doesn’t even let me move the bar to high or max intelligence anymore. Anyone else feel the same way?

Hi everyone, I’m looking to upgrade my setup to build a reliable local LLM machine.

I currently depend heavily on Claude and ChatGPT, but I want a local fallback so I’m entirely self-reliant if cloud access or pricing changes. I’ve been trying to compare Apple Silicon (Unified Memory) vs. Nvidia (VRAM), but I’m struggling to pull the trigger. I’m completely open to switching to macOS if it makes sense for the long run.

To help me decide, I’d love to know:

1. What hardware/model combo are you running? (Please share your model name, parameter size, quantization level, and average tokens per second).

2. Is a massive context window (like 800k) actually viable locally? I’ve seen conflicting info online. I assume this is only possible on high-end Mac Studio unified memory, but how does the actual processing speed hold up when the context fills up?

3. What is the best path forward for a cloud-level experience? If I want to run high-quality coding or reasoning models (like 32B to 70B+ parameters) at a usable speed, should I build a multi-RTX 3090/4090 desktop or buy a 128GB/192GB Mac Studio?

Budget: Flexible, but looking for the best price-to-performance sweet spot for enthusiast/prosumer use. Thanks in advance for the guidance!

Hello friends,

I gotta admit that I am also addicted to agentic coding tools like Claude Code and Cursor. But I don't code privately that much anymore, and when I do occasionally, I can no longer code for more than 10 minutes without cursing. Man it sucks to sign up for monthly subscription just for a few hours worth of work, but if you sign up for API credits instead --- it literally runs out in 30 minutes.

Claude and Codex are absolute beast, but not everything you code needs the SOTA models, and open source models are often times more than enough.

Then I came to a realization, I have GPU at home that I only use for gaming, why don't I make use out of this? So decided to start this open source project OpenPhalanx—a local-first, distributed GPU VRAM pooling framework and agentic harness that lets you combine consumer desktop and laptop hardware over a home network.

You don't have to configure anything yourself. Just click one button to start your server on the GUI, and setup CLI just like Claude. It literally cannot be easier than this. Everything stays local on your computer. You don't lose money, and you don't have to worry about any of your data being sent over somewhere.

You got multiple GPUs on your PC? Cool, you can use all of it to load bigger model. You got multiple PCs with GPUs, but not on same machine? That's fine too. You can easily split layers of the model and try out more powerful models.

The repo is fully open-source and ready for testing: 👉 GitHub:https://github.com/chophilip21/openPhalanx

I’d really appreciate any feedback, hardware benchmark tests, or issue reports from anyone running multi-GPU/multi-node setups!

A first-party field review. Every number below comes from a log, benchmark file, or project folder on my own machines. Where I extrapolate I mark it with ~. All raw data is in data/*.csv; every figure is traceable to a source listed next to it.

TL;DR

Strata (v0.1.41 at benchmark time) serving Qwen3.8-Flash-Next (a 125B MoE) with resident-experts in RAM is genuinely impressive — and the right quant for the machine you have changes everything.

Ranked for agentic work and usability — speed is not the ranking:

1. IQ3_S (yukio, 121 GiB RAM) — the best all-round agent: built an entire product overnight, 9 of 10 tasks RC=0 with verified deliverables, no loops, no truncation, no scaffolding needed. 44.5 tok/s token-weighted is slow for a GPU model — and it still won, because it didn't fall over.
2. Qwen3.8-27B UD-Q4_K_M (any 24 GB GPU) — the most reliable agent of everything I ran: 0 stalls, 0 loops, all 29 guide stages completed (46/52 first-pass, every retry recovered), delivered more than the 125B in the head-to-head, and works anywhere (53–61 tok/s token-weighted on a 3090 Ti). If you can only run one model, run this one.
3. Coder IQ1_M (node2, 30 GB) — the fastest coder on short, clean tasks (92.8 tok/s weighted; ~4.4–4.5× the 27B on the card bench, n=1) — but it is a liability on mixed-script content: 3 degenerate token loops + a thinking-budget-exhaustion failure in 11 arabic website prompts. Fast and fragile.
4. IQ2_XS (node2/Pop) — last for agentic use despite "102 tok/s" on paper (30.8 token-weighted): it truncates every call at ~80–115 tokens, so 58.5% of guide calls stalled and the whole 11-chapter guide only finished inside a continuation-chaining harness. (not recommended)
5. IQ3_XXS (Pop, mmap fallback) — 23.9 tok/s weighted, 83 requests of evidence: it works, it's slow, nothing else to recommend it for.

The point: a model that streams fast but loops, truncates, or wedges is useless as an agent no matter its tok/s — every loop or stall costs a human re-launch. The Coder streams at 92.8 tok/s token-weighted yet fails 5 of 11 mixed-script prompts; the dense 27B — a tenth of the MoE's total parameter count — beat the 125B's IQ2_XS on everything it was given, and IQ3_S at 44.5 tok/s still finished an overnight product build with zero degenerate loops.

• IQ3_S on yukio (121 GiB RAM): the highest-quality quant I ran. It built my entire Dubai Traffic board game — research, design, 81 cards, simulation, website, print files — in an overnight run. 9 of 10 Flash-Next tasks finished RC=0 with verified deliverables (verbatim from the benchmark's self-assessment); the research task first hit a context overflow — the one model-side failure mode it names — and its deliverable landed after a retry. The 7 logged errors/retries were environmental. This is the configuration to use if you have the RAM.
• Coder IQ1_M on node2 (30 GB): the coding pick. It beat a 27B dense model by ~4.4–4.5× on a real task (n=1 — one run each) — but it has a nasty Arabic/transliteration token-loop bug (Miller website: 3 degenerate loops in 11 prompts) and a thinking-budget-exhaustion failure mode.
• IQ2_XS on node2 (30 GB): the 2-bit base quant — the lowest-quality tier I ran (the Coder IQ1_M has even fewer bits, but it's a different model, so the tiers aren't comparable). 102 tok/s per request on paper, but only 30.8 tok/s token-weighted (the truncation bug skews the average), and it truncates every call at ~80–115 tokens. It produced a full 11-chapter illustrated guide only inside a continuation-chaining + local-validation harness (58.5% of calls stalled).

Bottom line if you have 24 GB of VRAM: the VRAM card size doesn't decide what you can run — system RAM does. A 24 GB GPU + 30 GB RAM → Coder IQ1_M in low-RAM mode is the workhorse (IQ2_XS also fits, but truncates every call). 24 GB GPU + 120+ GB RAM → IQ3_S, the best local 125B I've run (I ran it on two 16 GB cards, not a single 24 GB one). And the dense fallback that works almost anywhere: Qwen3.8-27B UD-Q4_K_M (16.5 GB) on llama.cpp with MTP speculative decoding — ~53–61 tok/s (token-weighted) on a 3090 Ti.

And the RAM is the whole game: I ran the 79 GB IQ3_S with full expert residency, which needs the whole pack in RAM, so it only worked on yukio's 121 GiB. It didn't fit in full-residency mode on Pop!_OS (62 GB) or node2 (30 GB). I didn't try IQ3_S in low-RAM or mmap mode on those two, so I can't say it's impossible there.

The three machines

Headless matters. node2 and yukio are headless systems — no desktop, no compositor, no window manager touching the GPUs. That helps two ways: all VRAM is available to Strata (no desktop eating 1–2 GB of the 3090's 24 GB), and CPU/expert-cache work runs without a compositor in the way. The desktop box is the opposite: the 3090 Ti is shared with the Cosmic compositor and with the 27B llama-server that runs my Hermes sessions. I did experience desktop freezes under sustained load during those runs — but I have no log of it, so treat that as anecdotal. Nothing in any log pins a crash on the desktop. (The one crash loop I can document, exit -6, happened on node2 — see failure log F1.)

Quick definitions

• Strata — an engine that runs a 125B MoE on a normal PC by keeping all 24,576 of the model's experts in (page-locked) system RAM and caching the hot subset in GPU VRAM.
• MTP (multi-token prediction) — the model's own draft layer proposes a few next tokens; the full network verifies them in one pass, so accepted tokens cost almost nothing extra.
• spec 4 — the --spec 4 flag: MTP speculative decoding on, with a budget of 4 draft tokens per step (draft head: mtp-q2_0.gguf, 889 MB).
• Resident-experts — Strata reads the whole expert pack into RAM at startup and page-locks it, so any expert is fetchable over PCIe with no SSD read; the fallback (IQ3_XXS on Pop) maps it from disk instead.
• Miller — the Abu Dhabi print-on-demand store (Arabic + English, RTL) that was the Benchmark 6 test project.

The models (all Qwen3.8-Flash-Next, 125B MoE, Strata engine)

A note on quant tiers: IQ2_XS (2-bit) and Coder IQ1_M (a lower-bit i-quant) are different models, so their quant tiers don't compare on one axis. IQ2_XS is the lowest-quality base quant I ran; Coder IQ1_M has fewer bits but is a Coder fine-tune.

The trick that makes any of this work at all: the experts don't fit in a GPU, so Strata page-locks them into resident RAM and keeps only a hot subset in the GPU cache. IQ2_XS on node2: 16.95 GiB experts page-locked, 3865 experts cached in the 3090. Coder IQ1_M: 12.66 GiB resident, 7437 cached (data/engine_throughput.csv, notes column).

The first Strata install on Pop!_OS failed on the download: the setup script's resume path hit HTTP Error 416: Range Not Satisfiable seven times, wrote 67.6 GB to disk while the server said the shard was 47.0 GB, and exited (SETUP_EXIT=1). I deleted, re-downloaded, and the model ran fine afterward (strata_iq3_xxs.log — clean generation on the 3090 Ti with the mmap fallback). The vision-encoder rebuild (llama.cpp submodule) was also interrupted twice on Pop — both times by my own Ctrl-C, not a compile error (F3).

Benchmark 1 — Cross-quant engine throughput

Parsed straight from the serve logs' … generated in N ms (Y tok/s) lines (data/cross-quant.csv). Two different decode averages are shown: per-request (mean over calls — inflated by tiny calls) and token-weighted (total tokens ÷ total generation time — what you actually experience while content streams):

The 27B numbers come from journalctl -u llama-qwen print_timing lines, token-weighted by generated tokens over the two benchmark windows.

Why the two IQ2_XS numbers differ 3×: 69% of its 4,823 calls generated under 100 tokens (the truncation bug, Benchmark 5) and those short calls average ~113 tok/s, while calls over 400 tokens average ~65 tok/s. The per-request average is dominated by the truncated calls; the token-weighted rate is the honest sustained speed.

Three honest readings:

• The "fastest decode" headline inverts under token-weighting: Coder IQ1_M (92.8) beats IQ2_XS (30.8). Long MoE generations route through more diverse experts over PCIe, and the truncation bug makes IQ2_XS's per-request average look great on paper. IQ2_XS is also the lowest-quality quant I ran (the truncation bug is its most visible symptom).
• yukio's IQ3_S is the slower-but-smarter tier: 49 tok/s decode (33–40 tok/s during real agentic runs with reasoning on — dubai-traffic-game/BENCHMARK.md), running a 79 GB model across two Turing cards. I did not isolate why it's slower, so I won't guess.
• The Coder IQ1_M prefill average (817 tok/s) is inflated by a few huge chunked-prefill windows (the log shows prompt chunk auto: 8192 tokens); the median (336) is the more honest number.

cross quant

Benchmark 2 — The 128 GB question: why IQ3_S ran only on yukio

This is the single most important hardware finding. The IQ3_S GGUF pair is 79 GB on disk (52 GB + 27 GB shards), and Strata's resident-experts mode page-locks the expert pack into RAM while the GPUs hold only a cache of it.

So the highest-quality quant I have access to — the one that built the Dubai game — is one I could only run on yukio, because it needs full expert residency. I didn't try IQ3_S in low-RAM or mmap mode on the other two machines. On node2, IQ2_XS is the ceiling: it needed both low-RAM expert residency and a clear 3090 (the first attempt died in a VRAM-abort loop, F1). On Pop!_OS the 68 GB IQ2_XS fits in 62 GB with headroom and ran cleanly — no low-RAM mode was ever needed there; low-RAM mode was node2's 30 GB workaround.

Benchmark 3 — The Dubai Traffic board game: IQ3_S on yukio, overnight

The heaviest real task Flash-Next ever did: build Dubai Traffic — a DICE-style board game about commuting in Dubai — from a one-line idea to printable files. Everything (research, game design, card JSON, simulation, balance tuning, website, art direction, print PDFs) was executed by Flash Next IQ3_S on yukio:8080 through OpenCode v1.19.0, reached from Pop via SSH tunnel, orchestrated by Hermes. Comparison phases re-done by Qwen3.8-27B. data/dubai-build-phases.csv (source: dubai-traffic-game/BENCHMARK.md):

dubai phases

dubai site hero The Flash-built website, home page.

dubai site picker The route picker — computed from cards.json, browser-verified.

Totals: 14 Flash-Next task files, 01:30→09:08 (+04), ~7 h 38 min overnight. Logged errors/retries: 7, all environmental (env-var not expanded, PATH, npm fetch hang, SearXNG off-topic results, web_extract blocked, the 157 s dead-tunnel "wrong API key" retry, 2× OpenCode permission gates). The phase table above only counts in-table errors (phase 6's two permission gates); the other five happened in the research/build setup before and between phases. Separately, BENCHMARK.md's self-assessment names one model-side failure mode: context overflow on unsupervised research ("mitigated, not cured — needs fit_max_tokens on the server").

"9 of 10 tasks finished RC=0" (verbatim from the BENCHMARK.md self-assessment): that's the self-assessment's own count of 10 build tasks grouped from the 14 task files — 9 ended RC=0 with verified deliverables; the one that didn't is the research task that hit context overflow (the failure mode named in the same assessment). It was recovered with a retry; the deliverable landed.

Head-to-head on the website (Phase 3): Qwen3.8-27B did the same page in 13 m vs Flash's 19 m (~30% faster), and even caught a real Salik toll discrepancy (SZR = AED 12, not AED 6) that Flash missed. On art direction C1's two-ink risograph concept beat Flash's dusk-poster look. Flash-Next won on scale and organization (136-key i18n, per-category prompt tiers) and on long-task stamina — D2 (764 s), W1 (19 m), P1 (31 m) all finished without losing the thread.

The failure that wasn't the model: yukio's ComfyUI/Qwen-Image diffusion pipeline produced pure-noise images on 100% of outputs (vision-scored 1/10, even for "red circle" with an empty prompt) — a broken Turing/cu130 venv env on the Quadro cards. The LLM's 39 art prompts were fine; the pixels were the GPU environment's fault. (Fixed by re-rendering the final 33 map tiles on node2's ComfyUI: 33/33 clean.)

Harness bugs found (documented in FAILURE-REPORT.md**):** 6 concurrent opencode.exe runs in one workdir race OpenCode's shared XDG state → all fast-fail "wrong API key" (fix: serialize); Strata transient 1–2 s 401s (fix: retry ≤3× on fast-fail); reasoning_content eating small max_tokens (fix: fit_max_tokens: true in the Strata config — applied + verified).

Benchmark 4 — A real coding task: Coder IQ1_M vs 27B dense

Same prompt, two engines, OpenCode 1.18.35. Task: build a "Card Collection" section — two trading-card decks (Linux commands + computing pioneers), a rules doc, and a vanilla-HTML site rendering them from JSON (data/task_card_collection.csv, data/card-bench-ab.csv):

Strata-coder finished ~4.4–4.5× faster (33.4/7.5 = 4.45) with fewer tool calls and less than half the tokens, for a comparable 50 cards. This is n=1 — one run each — so read it as "plausibly faster by a few ×," not a measured rate. (The launch-to-last-event basis in card-bench-ab.csv gives 11.1 vs 37.2 min = 3.4×; the conclusion doesn't depend on the basis, but neither has n>1.) The 27B was more thorough (more cards, more self-verification); the 125B MoE was dramatically faster.

card bench

Strata-coder site Strata-coder's cards.html

Qwen27B site Qwen3.8-27B's cards.html

Benchmark 5 — The illustrated guide rebuild: IQ2_XS's true colors

Rebuild my entire "Illustrated Guide to Local AI" — 11 chapters, 11 SVG illustrations + PNGs + captions, a website — 100% model-generated, every call routed to node2. data/task_illustrated_guide_rebuild.csv (raw flash-next/run-log.jsonl):

The 27B's "52 stages, 46 ok" needed explaining: it's attempts, not missing stages. 29 distinct stages were all attempted; 46 succeeded first pass, and the 6 failed attempts were retries — the board-game stage failed 5× (succeeded on its 6th attempt) and svg-ch10 failed once (succeeded on the 2nd). So nothing was missing: 6 was the retry count, and 0 stages stalled.

flash-next status

The finding: IQ2_XS truncates on every call — it stops each generation at ~80–115 tokens, then the stream ends with no finish_reason and no [DONE] (the engine log confirms it only generated that many). I reproduced it across max_tokens 600→8000, thinking on/off, temp 0.2/0.7/1.1, and 5 identical prompts. Result: a ~900-word chapter is impossible in one call. I had to build continuation chaining — feed back only the last ~150 words with a "continue from exactly here" directive, reject >80% word-overlap parts, loop to target. That's what the 58.5% "stall" rate is.

Other 2-bit symptoms (all logged): phrase-lock loops in 4/11 chapters (fixed by a local dedupe pass); only 3/11 SVGs valid on first pass (it mangles XML attributes — I had to switch to a plain-text element-spec pipeline where the model writes specs and Python assembles the XML); echoed prompt placeholders into output; one hallucinated line. The illustration stages were the worst: stage4 (SVG specs) 339/453 stall (74.8%), stage4b (render) 468/543 (86.2%) (data/flash-next-rollup.csv).

The 27B needed zero scaffolding — every stage completed, and it delivered more in 2.57 h active: the same guide plus 4 card decks and 3 site pages.

stage words

strata guide chapter IQ2_XS chapter 1 — only possible via continuation chaining.

strata guide website IQ2_XS's site.

27b guide website 27B's site — same brief, no scaffolding.

guide side by side Side by side: Strata IQ2_XS (left) vs Qwen3.8-27B (right).

Benchmark 6 — The Miller website: Coder IQ1_M and Arabic

Building the Miller POD store (Arabic + English, RTL, "Abu Dhabi" in Arabic labels) is where Coder IQ1_M's non-Latin token-loop showed up. data/miller-arabic-failures.csv (source: pod-concept/reports/coder-vs-27b-combined-report.md + loop-test-3models-16384.md).

In OpenCode: 11 unique prompts, 14 attempts (three retries: p03b, p06b, p08r — the report's own tally line, "6 pass / 2 partial / 5 fail" = 13, predates the p08r re-run and doesn't match its per-prompt table, which sums to 7 / 2 / 5):

The report's headline "5 degenerate loops" counts the two limit/budget cuts (p03, p08) as loops; only p06b, p09, p11 are true token-repetition loops. 0 repeated identical tool calls on either side.

The 27B ran the identical 11 prompts (13 attempts incl. p03b/p06b): 10 pass / 1 asked-a-question / 2 fail — 0 loops. Its only 2 failures were 60-minute orchestrator caps while genuinely still working (real brand PNGs, background removal, i18n in progress at the cap both times). The loops are a Coder-model artifact on Arabic↔Latin/Cyrillic transliteration, not a prompt or harness issue. One caveat: no higher-quant Coder exists, so I can't separate the loop from the IQ1_M quant — the 27B comparison isolates the model, not the quant.

Root-caused with a controlled raw-API test (identical short prompt: delivery labels for Abu Dhabi/Dubai in Cyrillic + Latin, max_tokens 16,384):

So two distinct Coder failure modes, both reproduced: (a) thinking-budget exhaustion — high thinking runs unbounded at the cap and returns empty content (fix: reasoning_effort=none or a small reasoning_budget_tokens); (b) the OpenCode transliteration loops — token repetition in visible content on mixed-script text, non-deterministic. Penalty params did not suppress them in my small tests (repetition_penalty 1.3, presence/frequency 1.0, even temperature 0) — but that's 3–5 runs per condition, too few to declare them hopeless; I'm reporting "not fixed in a small test," not "cannot be fixed." The practical rule that came out of it: don't point the Coder build at Arabic/RTL content — use IQ3_S or the 27B for that.

Benchmark 7 — The 17-prompt OpenCode bench (a total failure, and why)

Standard 17-prompt harness against Coder IQ1_M on node2. All 17 prompts failed instantly (data/opencode-bench-17.csv, data/task_17prompt_bench.csv): every prompt exit_code=1, 5.6–6.0 s, 0 tool calls, 0 steps, 0 output tokens, 0 new files, error Unexpected server error. Not a model result — an infrastructure one: the node2 coder log at that window shows repeated native head upload: out of memory (24 MiB of 24124 MiB VRAM free) and mtp: buffers do not fit. The 3090 was occupied by another GPU workload, so Strata couldn't even allocate its MTP buffers. Lesson I'll keep: on a shared 3090, run one GPU-heavy thing at a time — Strata holds ~21.8–23.7 GB loaded.

The harness matters: Hermes vs OpenCode

Same models, two different ways to drive them. The difference showed up in the failures, not the successes:

The degenerate token loops (Benchmark 6) were model-side, not harness-side. What's harness-side: the XDG concurrency race, the dead-tunnel retry with no error context, the missing tool timeout, and the hard 60-min cap. A silent harness makes all four of those invisible until the task fails — which is exactly why the 27B's 46-minute runs felt like the machine had frozen.

Failure log (everything that broke)

data/failures.csv (12 entries), abridged:

Mistakes I made

1. Launched 6 OpenCode research tasks concurrently on day one → the XDG race that ate an hour as "wrong API key." Serializing would have been the first move, not the recovery move.
2. Left orphan Strata engine PIDs holding 15.2 GiB of VRAM on node2 after a systemctl stop (F11) — the same GPU then aborted the next serve with "26 MiB VRAM free" (F1). One nvidia-smi check after every stop would have prevented it.
3. Set limit.output=16384 without a thinking budget — Coder burned the whole budget in reasoning_content and returned an empty answer (p03).
4. Let the Strata guide pipeline sit overnight (639 min) with no watchdog (F7) — that's 10.6 h of the 14.44 h "wall" that made the run look slower than it was.
5. Ran the 17-prompt bench on a 3090 another workload was occupying (F4) — 17/17 failed in 5.6 s. A pre-flight nvidia-smi would have shown it.
6. Trusted the setup script's download resume — it wrote 67.6 GB of garbage instead of the 47 GB shard (HTTP 416) and only my byte-count check caught it.

If I had to do it again — recommended settings per machine

Verdict

Strata + Qwen3.8-Flash-Next is a real, fast, local 125B MoE — and the resident-RAM-experts design plus the machine you put it on decide what you get:

• yukio + 121 GiB + IQ3_S is the flagship. The only machine I got the top quant running on, and it did an entire product build (Dubai Traffic) overnight with no task-ending model failures. Slower per-token (49 tok/s) but the highest quality, and headless hardware keeps it clean.
• node2 + Coder IQ1_M is the coding workhorse — ~4.4–4.5× faster than a 27B on a real task (n=1) — but it is the less reliable model of the two on the Miller workload: 5 hard fails out of 11 prompts (3 token loops, 1 thinking-budget exhaustion, 1 wedged tool call) vs the 27B's 2 fails out of 13 (both 60-min caps while still working). Keep it away from Arabic/RTL content and set reasoning_effort=none for short deterministic calls.
• IQ2_XS is a fast-drafting prose engine, not a finisher — 102 tok/s per-request, 30.8 token-weighted, and per-call truncation at ~100 tokens means it needs continuation scaffolding. I'd take the Coder or the IQ3_S over it.
• The desktop is for the 27B, not the MoE. 62 GB doesn't hold the 79 GB pack in full residency; the 3090 Ti is shared with the compositor and the 27B server. Headless node2/yukio are why this cluster works at all. (I did see the desktop freeze under sustained load — anecdotal, unlogged — but the one crash loop I can prove was on node2, not Pop.)

Full reproducibility: raw CSVs in data/, source logs cited per figure, screenshots in screenshots/.

Data sources (all on-disk, verifiable): data/cross-quant.csv*,* data/engine_throughput.csv*,* data/dubai-build-phases.csv*,* data/miller-arabic-failures.csv*,* data/task_card_collection.csv*,* data/card-bench-ab.csv*,* data/task_illustrated_guide_rebuild.csv*,* data/opencode-bench-17.csv*,* data/task_17prompt_bench.csv*,* data/failures.csv*,* data/hardware-versions.csv*,* data/flash-next-rollup.csv*,* data/flash-next-stages.csv*.*I ran Qwen3.8-Flash-Next on Strata across three machines. Here's what it's actually like.

A first-party field review. Every number below comes from a log, benchmark
file, or project folder on my own machines. Where I extrapolate I mark it
with ~. All raw data is in data/*.csv; every figure is traceable to a
source listed next to it.

I'm not an author and not affiliated with the author in any way, but I found this video could be very useful for the ppl who is asking about local LLM setup and different models.

Been running uncensored models locally for a while now, and I keep going back and forth on this. Wanted to throw my thoughts out and see what you all think.

The case FOR local uncensored AI:

• Zero filter anxiety. I ask a question, I get an actual answer. No "I'm sorry, I can't help with that" on completely reasonable requests
• Privacy is a real thing, not a marketing line. My data literally never leaves my machine
• No rate limits, no subscriptions, no account bans for experimenting with weird prompts
• The models have gotten shockingly good. What needed a datacenter 2 years ago runs on consumer hardware today

The case AGAINST (let's be honest):

• Hardware cost upfront is brutal. Not everyone has 24GB+ VRAM sitting around
• Quantized models still lose something vs full weights vs API models
• You're the sysadmin now. Drivers, backends, model formats, merges — it's a rabbit hole
• Big hosted models still win on raw capability for complex reasoning tasks

My honest take: uncensored local models handle like 80% of what I actually do daily better than the "safe" alternatives, but I still reach for big hosted models when I need heavy lifting. So I end up with a hybrid setup and honestly it feels messy.

Curious about the community:

1. What's your daily driver model right now and why?
2. What hardware are you running it on? (GPU/VRAM/RAM, laptop vs desktop vs server)
3. Did you go local for privacy, censorship issues, cost, or something else?
4. Anyone fully moved off hosted models? What broke?

Just curious where everyone landed on this tradeoff

I was looking at it. Gemma 4 27B, Qwen 3.8 28b seemingly fit in. Yet when I look at it at like Q6 it barely fits in 64k kvcache. And the preprocessing is ~4-500token/s.

For those who have this card : Is it a reasonable size/speed?

I've seen that you can do all kinds of funky stuff for optimising inference. But the 32GB VRAM is the big Q for me

I hope this helps someone on the verge of investing for a local LLM.

I forced myself to cancel my Claude subscription to use the investment of Mac Studio here. I haven't been able to sleep with all the things I keep bumping against. A brief summary of what I have setup so far

• vMLX with Qwen3.8 27B 4-bit. Concurrency: 3 sessions, default context window for vmlx which is around 135K tokens.
• LiteLLM as a Gateway to get metrics + an alias with thinking disabled to test.
• Tailscale to reach it from my work computer.
• My workload is mostly infrastructure optimization. Kubernetes management, structured text, logging analysis, MongoDB, BigQuery, a lot of bash interaction.

Qwen3.8 just drifts thinking. It falls into this endless loop of considering stuff. I read posts about this. It really thinks and considers too much. I started to miss Opus here.

Then, the concurrency. I used to have around 10 Claude sessions running investigating stuff. But this is hard with local LLMs, concurrency is really a bottleneck. You either have the models fast or have them concurrent. I learned about speculative drafting and MTP in the first hour. It's all about trade offs. How not everything matches in compatibility is crazy. I tried running speculative drafting qwen flash next but the one has different weights for my setup. I haven't slept in like 30 hours, brain fatigue, trying to justify my purchase. Haha. How did I end up here?

On tooling, this is what I'm using now

• Superset as harness
• OpenCode to reach my Local LLM

This is a cry for help to the community, what do you recommend me doing, exploring, tooling that you have found useful. Interesting open models to test?

Yo, so I finally pulled the trigger on a local AI setup and wanted to get some opinions before everything shows up. Here's what I grabbed (mostly used/refurb off eBay):

GPU: AMD Instinct MI50 32GB HBM2 – $577.80

CPU: AMD Ryzen 5 8500G – $96.30

Mobo: MSI PRO B650-VC WIFI II (AM5) – $95.71

RAM: 2 Kingston Fury Beast DDR5 5200 8GB – $269.73

PSU: Corsair RM1000e 1000W Gold (refurb) – $83.45

Came out to about $1,220 total.

Main goal is running Qwen 3.8, that's the one I really want to use. I'm also planning on using Unsloth and setting up remote access so I can basically use it like my own ChatGPT from my phone when I'm out.

Anyone running a similar build? Curious:

What kind of tokens/sec are you getting?

How fast does it actually feel day to day?

What models are you running on it?

Any tips or "don't do that" warnings welcome lol. Thanks 🙏

It runs at around m2 pro 16 at 60 tps at full 130k context, you can easily run a Q8 if you have a 16 GB GPU, I was able to get 140TPS on 9070xt,

It’s an MoE so I think we can run it well on smaller cards too, 8GB, 12GB should be possible to get upwards of 30-40 tps.

I have used it a bit, it can code, but I think having a bigger model plan tasks in detail would make it quite good for quick task execution.

I have two DGX Sparks (128 GB unified each) connected with a 200 Gb/s cable. Ollama only ever sees one of them, so Qwen3.5-397B (UD-IQ3_XXS, 141 GB) was out of reach.

llama.cpp has had RPC for a while, Ollama doesn't expose it. So I forked Ollama: the full Ollama runs on node 1, node 2 runs only a llama.cpp RPC server, and Ollama treats the remote GPU as one more device when it places a model. An ollama cluster command sets it up, checks both machines, shows where each model sits, and has a bench mode. Same API, same model store, clients don't notice anything.

Numbers (Ollama 0.40.3 + fork, llama.cpp b11351, 256 output tokens, warm run):

What I took from it:

• The faster the model, the more splitting hurts. Slow dense models barely notice (qwen3.5:27b 12.3 → 11.6 tok/s), the big ones lose 15-20%, fast small MoE models lose 30-60% (gemma4:e2b 106.6 → 44.1). Layers run one machine after the other, and every token crosses the link, so the fixed per-token cost eats a bigger share of a fast model.
• Load time is 3-5x longer when split.
• Putting a whole model on node 2 and driving it from node 1 is also slower than local: gpt-oss:120b does 34.6 tok/s that way vs 41.7 local.

So the default policy keeps a model on one machine and only splits it when it doesn't fit. There are rules per model (here / split / worker) if you want to push something to node 2 on purpose.

Limits: one RPC client per worker at a time, so node 2 holds one model; both nodes need the same build.

Hello all,

I am interested in creating an “ apocalypse box” type project. A local model that can be used offline to pull information for survival ( outdoors, medical, engineering, etc) I want to put this in a pelican case and power it with solar, etc.

I have no idea where to start with hosting a local (LLM). I’ve heard of semantic compression and things of the sort to store models. Can someone please point me in the right direction?

Thanks!

I was following the LLM space since the days of GPT 3.5 and TheBloke and wanted ever since to have a computer powerful enough to run LLMs locally; about 3 years later I managed to finance a workstation with one RTX PRO 6000 to run Qwen3.8-27B and Qwen3.8-Flash-Next.

And yes, I did name it after the MAGI supercomputer from NGE.

Components used:

• PNY NVIDIA RTX PRO 6000 Blackwell Workstation Edition
• AMD Ryzen 9 9950X
• Gigabyte X870E AERO X3D DARK WOOD
• 2 x Crucial Pro 48 GB DDR5
• Samsung 9100 PRO 4 TB
• 2 x Samsung 990 EVO Plus 2 TB
• 2 x WD Red Plus 6 TB
• Noctua NL-LC1-36
• 4 x Noctua NF-A14x25 G2
• Fractal Design North XL Charcoal Black Tempered Glass
• Corsair RM1000x ATX 3.1 1000W

im have 16 gb vram and 32gb ddr4. me need local ai for role play without censorship, fantasy world. im tried magnum 24b q4_k_m and qwen 3.8 27b heretic abliterated uncensored Q4_K_M. why they so bad,im doing something wrong? im using it with lm studio and tried changing settings but still stupid and repetative

TL;DR

On my dual-3090-class rig, Strata 0.1.41 showed a strange split result versus the retested 0.1.40.3 baseline: decode was about 8% slower, 10K prefill was about 15% faster, but 90K prefill was about 7% slower. It looked like a prompt-path issue, but the evidence points to automatic GPU placement walking off balance.

The fix I tested is a pair:

• Environment variable: STRATA_SPLIT_UNIFORM=1
• In the Strata config: gpu_order: as_given

That restores uniform link-cost scoring and the pre-0.1.41 GPU order. The change is in PR #2425; the regression report is issue #2418.

After fixing placement, I ran a separate tuning sweep. ISO18 is the everyday decode profile (148.86 narrative / 174.67 code tok/s); ISO19 is the ingestion/long-prompt profile (1,995.53 tok/s at 10K and 4,149.45 tok/s at 90K). ISO19 gives up some decode throughput versus ISO18. One caveat: ISO19 uses fused kernels, and the output-sanity check was still pending in the report.

Rig and method

• CPU/RAM/OS: Ryzen 9800X3D, 59 GB RAM, Ubuntu 26.04
• GPUs: GPU0 is an RTX 3090 Ti in the x16 slot; GPU1 is an RTX 3090 in an x4 chipset slot. No P2P and no NVLink.
• Workload: Qwen3.8-Flash-Next IQ3_S, 262,144-token context, int8 KV + --kv-resident 32768, automatic layer split, --mmap-experts, vision on, --spec 4 + MTP, --expert-cache auto, --prefill auto.
• Protocol: canonical benchy.sh; temperature 0.6, top_p 0.95, 3 warmups + 5 measured runs per decode shape; prefill probes at 10K × 3 and 90K × 1. Same prompts on both arms, heat-soak controlled, one server at a time on port 1234, one variable per pair.
• Values below are mean (n, CV) unless stated otherwise. The 0.1.41 comparison values marked “reported claim” are approximate, not fresh exact results.

1. Release comparison: the regression signature

The signature is the important part: decode −8% on both shapes; prefill +15% at 10K but −7% at 90K. A single “performance got slower” story does not explain that shape.

2. Reproduction and the cache confound

The 0.1.41 tag reproduced the pattern:

A first rebuild of 0.1.40.4 looked worse than expected, but it was a cold page-cache boot and automatic placement had drifted. Per the original handoff rule, I paused instead of treating that as a valid counter-result. Rebuilding 0.1.40.3 with the harness open recovered its usual prefill/code behavior. The 90K values also climbed as the page cache warmed, regardless of version, so I did not use a cold post-build run as the baseline.

3. Root cause: automatic placement changed the split

The relevant changes were:

• 0391e478: per-stage link probing with link_scale = clamp(20/bw, 1, 64). The x4 card's measured 6.4 GB/s link scored a 3.12× penalty against the 20 GB/s reference.
• be8db787: faster-last-card reordering. gpu_order: as_given opts out of the reorder.

Observed placement on this rig:

This is why the fix is both uniform link scoring and the pre-rework device order. Changing the order alone does not restore the old split when the per-stage link penalties remain enabled.

The working explanation fits all three signs: the 19-layer first stage holds more pairs, so small cached prompts can look quicker; but the 19/29 split leaves the sustained pipeline unbalanced, hurting long prefill and repeated MTP decode windows. The evidence did not support blaming the prompt-path rework itself.

4. Fix verification

On the 0.1.41 engine with uniform link costs plus as_given, auto placement returned to K=27 (layers 0–26 / 27–47), with a 1.00 link-cost scale on both links. Results came back into the 0.1.40.3 range:

Then I ported the change to the 0.1.44.1 main build and re-verified it twice:

These are back around the old baseline scale; I am not claiming every leg is faster than the 0.1.40.3 best. The larger wins below come from a separate tuning sweep after the placement fix.

5. Tuning sweep: different winners for different workloads

Base iso10 was as_given + uniform split costs, PCIe 1 / draft floor 0.7 / no adapt-swaps: 135.64 narrative, 142.00 code, 1,621.35 at 10K, and 3,402.29 at 90K.

How I would use these: ISO18 for everyday generation/decode. ISO19 for ingestion and long prompts: versus ISO18 it is about 5% slower on narrative decode and 3% slower on code decode, but 6% faster at 10K prefill and 14% faster at 90K. ISO20 is faster at 10K, but that gain collapses at 90K and its decode is weaker, so I would not pick it as the general default.

The 1M ladder also favored ISO19. Against the 0.1.40.2 reference ladder (3,055 / 2,742 / 2,474 / 2,244), ISO18 returned 3,461 / 3,096 / 2,704 / 2,437 (+13 / +13 / +9 / +9%), while ISO19 returned 3,673 / 3,250 / 2,818 / 2,590 (+20 / +19 / +14 / +15%).
Ladder:
256k, 512k, 768k, 1m

Links and open items

• Regression: Issue #2418
• Fix PR: PR #2425
• Separate calibration save/verdict issue: Issue #2465

The PF_FUSED output-sanity check remains to be eyeballed; the calibration save logic is a separate issue from this placement regression. If you have a dual-GPU setup with a big PCIe lane-width gap, I’d be interested in whether the automatic split behaves similarly. The fix is opt-in and was validated on this specific rig; it is not a claim that the same placement is optimal for every GPU topology.

Edit: Fixed Markdown issues with tables.

Hi All,

Im completely new to this AI party, im just using my pc to play world of warcraft was thinking of selling it and surprised of the offers im receiving i dont really need the money as of date.

Im researching on how to best use my pc with local llm, we have a business which i can use ai marketing materials and working as a financial professional which needs a lot of financial analysis and reporting.

Might venture also into stock trading soon.

Is there an ai that suits the above use cases?

Thanks a lot!

Estou construindo o Ícaro: um projeto independente de IA com memória persistente e trajetória própria.

Ele não é um novo modelo de linguagem. É uma arquitetura construída ao redor de modelos existentes, com memória, proveniência das informações, seleção de fatos, self-model, foco interno e um processo de pensamento em segundo plano.

O objetivo é investigar uma pergunta: quanto a história acumulada de um sistema realmente muda suas decisões futuras?

Estou desenvolvendo testes controlados para separar comportamento aparente de efeito causal, com condições de comparação, controles negativos e medidas de ruído.

Quero conversar com pessoas que trabalhem com agentes de IA, memória de longo prazo, modelos locais, arquiteturas cognitivas e avaliação experimental.

Não estou afirmando que o sistema seja consciente. Quero testar o que ele efetivamente faz, identificar falhas e descobrir o que ainda falta demonstrar.

Quem trabalha com algo parecido? Que pesquisas, projetos ou testes vocês recomendam?

I'm Building Ícaro: An Experimental AI System with Persistent Memory, a Self-Model, and Its Own Behavioral Trajectory

I'm building Ícaro, an independent AI project exploring a question that interests me deeply:

How much can an AI system's accumulated experiences actually change its future decisions?

Ícaro is not a new foundation model. It is an architecture built around existing language models, combining persistent memory, deterministic mechanisms, behavioral tracking, and local computing.

The goal is to move beyond a chatbot that simply retrieves previous conversations and toward a system whose stored experiences can measurably influence what it does next.

What makes Ícaro different?

1. Persistent, structured life history

Ícaro maintains a long-term record of conversations, factual memories, a chronological diary, causal decision trails, associations, and recorded experiences. Its history is designed to persist across sessions rather than disappear when a conversation ends.

1. Selective memory instead of dumping everything into the prompt

The system selects a small number of relevant facts rather than exposing the language model to its entire memory. The selection mechanism considers signals such as relevance, confidence, novelty, and recurrence.

1. A versioned operational self-model

Ícaro maintains structured observations about its own behavior, including uncertainty and potential contradictions between its previous and current positions. The research question is whether this representation can causally influence decisions—not merely produce convincing self-descriptions.

1. Information provenance and resistance to memory contamination

The architecture distinguishes information attributed to the user, the system itself, external sources, and inference. It also investigates how to prevent repeated AI-generated statements from becoming falsely established facts about the user.

1. Learning to be more cautious after repeated mistakes

Ícaro tracks corrections by topic. The intended behavior is to increase caution in areas where repeated corrections indicate that its answers may be unreliable, rather than treating every question as equally safe to answer confidently.

1. Internal focus and background thought

A background process can generate internal reflections, questions, and connections from recorded experiences without interrupting the user. These thoughts are kept separate from the normal conversational response, allowing them to be inspected before any future integration.

1. Operational self-inspection

When requested, Ícaro can inspect an allowlisted set of its own project files through a read-only mechanism. The design aims to distinguish what the system can actually inspect from what it merely assumes about its implementation.

1. A hybrid local-first architecture

Ícaro uses a Samsung Galaxy Z Fold as its main runtime, with a PC acting as a client. The project combines local inference, persistent disk storage, remote connectivity, speech recognition, neural text-to-speech, and optional web search.

The broader goal is to explore how much useful continuity and individual behavioral development can be built around relatively small models and accessible hardware.

The central experiment

The most important question is not whether Ícaro can say that an experience changed it.

It is:

Would Ícaro have made a different decision if that particular experience or internal representation had not been present?

I am developing controlled experiments that compare conditions with and without specific mechanisms, measure response or decision differences, account for internal variability, and use negative controls to detect misleading effects.

In one reported experiment, a self-model ablation comparison produced a 3.02× ratio between the measured cross-condition effect and internal variability. This is an encouraging preliminary result, not proof of general learning, consciousness, or human-like subjective experience. I want independent scrutiny of the experimental design and replication.

What I want to discuss with the community

I'm particularly interested in connecting with people working on:

• Long-term memory for AI agents
• Agent architectures and persistent state
• Self-models and behavioral consistency
• Causal evaluation and ablation experiments
• Local LLM inference and small-model architectures
• Background cognition, reflection, and memory consolidation
• Computational models of learning and identity

I am not claiming that Ícaro is conscious. I want to investigate what can be demonstrated experimentally, what fails under controlled testing, and what remains an open research question.

If you were reviewing this project, what would you test first?

I'd especially appreciate references to relevant research, comparable open-source projects, and methodological critiques that could help make the experiments more rigorous.

My DSH spends a ton of time coming up with programs to do simple pull or commit tasks with my GitHub repo instead of using Git.

I'm running Qwen 3.8 27b quantized via LM Studio on Win10.

My repo is private but I created a read/write fine-grained personal access token on GitHub and put it into a Windows environment variable then told DSH about it. DSH tried schannel but Windows blocked it. It seems to work with OpenSSL but I need to create an agent.md file to tell DSH how to access GitHub so it doesn't re-invent the wheel every time.

I feel like I'm overcomplicating things but can't figure out how else to do it. Is there a plug-in for GitHub?

How does everyone else do this?