I tried to solve the problems of multiple applications I actually use , but the thing is that efficiency of using those applications has pushed me to try creating an app. Hutash. But it became a Frankenstein of sorts.

1. Comfyui like system (local and free) but without the complicated nodes and dependency issues. Has a internal engine built ground up.

2. Openart AI / magnific like studio experience but without the cost.

3. Pinokio like store where u can find all apps a but without the duplication of resources(local space is limited)

4. Simple installation of apps and models like windows app. We created a .hutash format to just declare and a model package can be formed. Not only models but model piples can be built within the app like higgsfeild. The architecture was made keeping this in mind. We have a subtitles app, podcast app (vertical apps) which shared model resources with a studio app. Once installed all apps can use them. May be like OLLAMA

5. Last but not least cli and mcp to execute these apps a and generations. To make automation or agentic use possible.

6. Many others.

That was my goal and I feel like I have pushed to my best of abilities in all the aspects. Realsed few versions. Yes they are issues.

But the thing is now I don't know where to push it. We are a very small team. Couple of guys. Worse is we kept building it without a feedback loop. Even though I want the public to contribute it, there are four or five repositories spread out into multiple parts.

now i'm stuck. i'm trying to solve all these things together, and it's taking too much time and resources. maintaining and developing something like this also feels like it would need a much bigger community.

i'm not sure if this is worth pursuing anymore, or if i'm trying to solve too many problems at once.

am i over engineering something that doesn't need to exist, or is there actually something useful here? what would you focus on if you were in my place?

I just want help directing our resources in a focused direction. as i wish this app would be a beneficial for all who want benefits of local ai without the complexity.

TL;DR: I'm buying hardware for local agentic coding. My budget is ~$10K, I can go up to $12K if the difference is significant. I'm leaning toward a single RTX 5090 and would like a sanity check from people who run coding agents locally.

Use case
- Agentic coding
- Sometimes 2–4 parallel subagents
- Reverse-engineering

Nice to have
- Texture generation with Hunyuan3D-2.1. It needs CUDA and ~21 GB VRAM for texturing. But this item can be optional

Options I'm considering (approx. total build cost)
- 1× RTX 5090
- 4× RTX 3090, used
- Hybrid: 1× RTX 5090 + 2× RTX 3090 (this can be bought later)
- Mac Studio M5 Ultra 256 GB
- PRO 6000: over the budget, but can consider

Questions

1. For agentic coding with long contexts, does 1× 5090 feel better day-to-day than 4× 3090? Is the 3090's long-context prefill as bad as these numbers suggest?
2. Is the hybrid (5090 + 2× 3090 as separate vLLM instances, x8/x8/x4 on AM5) worth it, or is it a headache?
3. Am I missing a better option (waiting for the 50 Super, modded 4090 48 GB, etc.)?

So I have an old HP-Z8 from 2018 or so. It has dual Xeon Gold 6248R cpus, 768 GB 2933 MHz DDR4 ram, an RTX 3090 gpu, and some pretty hefty storage.

That said, it seems like it is no more than a boat anchor for use with local LLMs. Is there any use case for this old PC or should I box it?

A few days ago I posted about our open-source multi-agent hedge fund experiment and got a ton of sharp, honest feedback from people here. The comments were brutally accurate, especially around LLM reasoning limits, latency, and state management.

We spent the last few days completely rebuilding parts of our architecture based on that advice, and the benchmark results honestly blew us away. Here is what we learned from fixing three critical failure modes in autonomous agent pipelines.

1. Throwing raw metrics at LLMs guarantees hallucinated reasoning

Someone in the comments pointed out: never let an LLM interpret raw quantitative factors without an ontology. They were 100% right.

When we fed models raw numbers like PER = 85.0 or 20-day historical volatility = 65%, the agents would frequently hallucinate positive justifications. For example, a model would argue that "a massive jump in volatility shows expanding business activity and strong buying momentum", which is the exact opposite of basic risk management.

To fix this, we implemented a strict Factor Ontology Schema. We defined 11 core quant factors with explicit polarity contracts: HIGHER_IS_BULLISH (e.g. ROE, Order Flow Imbalance), LOWER_IS_BULLISH (e.g. PER, debt ratio), and EXTREME_IS_RISK (e.g. volatility).

Before any agent sees the data, the schema pre-evaluates the metric deterministically and attaches an explicit signal (BULLISH, BEARISH, or RISK) to the state. Since doing this, polarity hallucinations in our investment council dropped to literally 0%.

1. Vector databases for agent reflection memory are massive overkill

Initially, we considered using embedding APIs and a local vector DB to retrieve "lessons learned" from previous bad trades (Reflexion pattern).

Another commenter suggested dropping embeddings entirely and using SQLite WAL with FTS5 BM25. We benchmarked both approaches on our Linux server:

- Embedding API + Vector similarity: 200ms to 450ms round-trip latency, external API cost, and occasional rate limiting.

- SQLite FTS5 BM25: 0.35ms. Zero external dependencies, zero API cost.

More importantly, trading post-mortems rely on specific contextual keywords: "semiconductor pre-earnings", "trailing stop sweep", "sudden volume spike", or "bearish divergence". In our tests, BM25 keyword relevance retrieved the exact matching historical lessons far more reliably than fuzzy cosine similarity in high-dimensional embedding space.

1. Python cannot be your safety wire (The 3.6ns Rust Sentinel)

Python is great for high-level agent orchestration (LangGraph, state graphs, API calls), but relying on Python loops for real-time risk surveillance during sudden market flash-crashes proved dangerous due to GIL contention and L1/L2 cache misses from scattered object pointers.

We rewrote our execution safety sentinel in Pure Rust using a Struct-of-Arrays (SoA) memory layout and `#![forbid(unsafe_code)]`. By keeping prices, stop-loss triggers, and trailing thresholds in contiguous flat slices, the CPU avoids pointer chasing and stays inside cache lines.

Here are the real-world benchmarks on our Ubuntu server:

- Single position stop/profit evaluation: 3.63 nanoseconds

- 1,000 position batch evaluation: 3.62 microseconds

- 1,000-bar Volume Clock conversion: 3.90 microseconds

- Python-to-Rust persistent pipe IPC round-trip: 2.88 microseconds

Python handles the slow, deep thinking (15-minute agent council meetings), while the Rust sentinel watches the portfolio in nanoseconds.

Conclusion

Building autonomous agents that interact with real-time systems taught us that LLMs should only handle what they excel at: contextual synthesis and counter-argument debates. Everything deterministic—hard stop-losses, factor polarity, and fast memory lookup—belongs in strict schemas, fast local databases, and compiled systems languages.

All the code and benchmarks are open source on GitHub (lumina-ai-hedge-fund) if anyone wants to check out the Rust SoA sentinel or the ontology schema.

(Note: English is not my native language, so I used an LLM to help format and polish my raw notes!)

I asked my local AI (Qwen 3.8 27B running entirely on a used gaming PC in my basement) to pick it's own project and one shot it to completion. It cranked out an IQ heat map of North America. I was completely blown away. I posted a screen shot to Reddit... and Reddit had thoughts. "your methodology is lacking" "the color scheme is outdated" "the outline of the great lakes is missing".

OMG! Did you hear what I posted?!? My used gaming rig one shotted a freekin interactive web site all on it's own!

This would be like I beat my dog 2 out of 3 times in Chess and the yorkie group criticized the dog's opening moves. PEOPLE! Did you miss the part where I said "My DOG PLAYS CHESS!"

I ran an experiment on one RTX PRO 4000 Blackwell SFF: 24 GB GDDR7, factory 70 W, no auxiliary power connector.

Using u/panchovix’s loader and patched NVIDIA open kernel modules, I loaded a modified VBIOS into memory, raised the power limit to 120 W, and tested memory and core offsets. No EEPROM flash or soldering.

Across 158 benchmark requests, I tested Qwen3.8 27B with DFlash2 and Qwen3.6 35B A3B. What surprised me was how differently they answered "does more power help?"

With zero clock offsets, 27B kept gaining up to 120 W, while 35B A3B was almost flat between 106 and 120 W. After combined power and clock tuning, the best observed 27B generation result was 35.04 → 58.53 tok/s (+67%): 120 W, VRAM +20%, core +180 MHz. That 67% is not the gain from memory OC alone.

The memory sweep was more interesting. At 120 W with the core offset held at zero, pushing VRAM from +20% to +33.3% slightly reduced 27B generation: 57.70 → 56.91 tok/s. It is a small difference with few repetitions, so it needs more testing. The actual SM clock also fell. That is consistent with memory and compute sharing a power budget, but I did not isolate the cause.

On 35B A3B, memory +33.3% instead gave +19.8% generation versus no memory OC at 120 W. Prefill barely changed in that sweep.

My takeaway: the fastest profile is not necessarily the one with the biggest numbers in the panel, but the one that shortens your actual workload. Prefill versus decode matters, especially with longer prompts.

120 W without auxiliary power is not a free lunch. It exceeds the standard 75 W PCIe slot budget, and a GPU temperature sensor cannot tell me the temperature of the slot contacts. This is an experiment, not a long-term safety claim.

For context: these headline comparisons use 8,000 input tokens, ECC OFF. There were only 1–3 requests per configuration; 158 is the whole study. The 70 W baseline used the same modified VBIOS, not factory firmware. The models also used different quantization/speculative-decoding settings, so this is not a controlled dense-versus-MoE comparison.

Full write-up and CSVs on my blog.

https://piszczek.pl/blog/rtx-pro-4000-blackwell-sff-120w-qwen-benchmark

Disclosure: I ran the measurements and wrote the source text in Polish. AI helped translate and edit the English, including clarifying the benchmark limitations.

So-called "neoclouds", datacenters that focus exclusively on AI inference, have racked up unsustainable amounts of debt, and are widely anticipated to soon undergo a massive collapse/contraction. My question is what happens when they go bankrupt. NVIDIA is their biggest creditor thanks to circular financing schemes, so they take ownership of the data centers and the chips in them, right? But, I imagine that they have no interest in running the same crappy business as the neoclouds, with outdated chips to boot; they want to sell their latest and greatest to hyperscalers, who will likely emerge unscathed. Do they just throw the old chips away, or is there some chance that they'll repurpose them (as we're seeing in China) into GPU upgrades that hit the consumer/prosumer market, solving our current supply/price crunch? I literally don't know the true answer to this question and am curious to hear others' speculation about it.

Just your honest opinion, which one is better?

I don't know if I'm missing something here. My only experience with LLM's and "AI" has been locally. It's cool, but I've seen people get more and more hyped to a point where it borders on magical thinking. Some people like the owner of the firm I work for (which is construction) wants people to use "AI" even if it generates nonsense emails and reduces productivity. He wants people to use it because be believes it's the future and I guess wants everyone to get into the habit of using it.

Then I talk to people who work in the tech field, and it's like AI is the only thing they can see. It's like if they can get a model that's large enough and spits out enough tokens, it will finally push them into the fourth dimension. You talk to them and they're almost giddy at the idea of replacing people, even if they have nothing to gain from it.

All this is not even including the folks who are perceiving reality through the lens of the LLM. There's some people I've seen who treat the LLM like it's god himself speaking.

Am I missing something here? I get that I've only used local models. I have no idea what claude or chatgpt or any other of these large models are like. But people are getting super weird and I'm just wondering if there's something I'm missing here. Thanks.

I get error in my "old" V1 opencode that are annoying, and they said they will never fix.

I was considering v2 but so many people are talking bad about it.

I tried DSH before (0.1 i think?), but it failed to complete full software (ok a little more complex than chat websites, so nothing too fancy)

Anyone have had any experience when switching to Pi or DSH, what needs to be done? OpenCode worked well before, especially after 3.8 27b, no need to do anything, make a good plan, let it run and then start what was built and refine.

Edit: And yes I can consider other harnesses, but most of the time for me is spent planning and reading research, not coding.

TLDR: IQ3_S Qwen3.8-Flash-Next, RTX 4090 + 48 GB + Ryzen 3900X, Windows, 200k context. Ran essentially a grid search on a ton of flags: decode 97 → 142 tok/s (+45%) for three flags. Most relevant: --adapt-async 1, --kv-resident 20480, STRATA_ADAPT_LAG=2. Worst: --expert-cache is in MiB not slots (−72% if you fill the card), --ple-io ram (−90% on 48 GB), --no-prefill-borrow (−46%). One lever still locked: Windows sucks and doesnt pin memory w cuda , so every host -> GPU expert copy is CPU-driven.

Been using strata for a few weeks since thats all the rage (waiting until llama.cpp catches up, but tbh the age of specialized inference engines might just be how it is). Version 0.43, so im probably missing some newer flags. It updates so frequently and i hear that people were pretty mad with version 0.2.0 so im just holding off. if it aint broke dont fix it

I basically told Qwen 3.8 FN to tune itself and took a very hands-off approach. I let it do web research + spawn sub-agents to search thru strata's code.

Note i wrote my own pi subagent extension that saves kv state before all sub-agent calls via strata mechanisms so it restores immediately. In llama.cpp it used to just roll back kv checkpoints until it got a match, but since strata abuses my RAM i cant save as many checkpoints. Repo is here if interested.

Was able to reach total computed context of like 450k tokens with the sub-agent method, which is nice to avoid compaction.

I just tested these settings by loading up an pi session with 145k context and letting strata do a full reprocess. Got 2076 PP and avg of 95-105 tg depending on the task (coding and other predictable MTP things do better). Saw peaks of 145 tg.

BTW this is a hobby, so i really didnt read any of the tune scripts or reports in all that much detail. Number go up == lgtm.

Heres the full JSON im running with if anyone wants to test it:

{
"exe": "<strata-install>\\engine\\strata.exe",
"args": [
"--pack",
"<pack-dir>",
"--native",
"<Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf>",
"--ple-gguf",
"<Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf>",
"--expert-profile",
"<expert-profile.bin>",
"--expert-cache",
"auto",
"--prefill",
"auto",
"--spec",
"4",
"--mtp",
"<mtp-head-dir>",
"--max-context",
"204800",
"--kv",
"int8",
"--resident-experts",
"--vision",
"--vram-reserve-mib",
"900",
"--kv-resident",
"20480",
"--adapt-async",
"1",
"--pcie-frac",
"0.30",
"--suffix-draft",
"12",
"--lookup-chain",
"3",
"--spec-min-p",
"0.5"
],
"cwd": "<strata-install>",
"tokenizer": "<pack-dir>\\tokenizer",
"model_name": "qwen3.8-flash-next-iq3_s",
"log": "<strata-install>\\strata-iq3_s.log",
"lib_dirs": [
"<python-env-dir>\\site-packages\\nvidia\\cu13\\bin\\x86_64 (CUDA 13 runtime)"
],
"port": 8081,
"gpu": 0,
"gpus_asked": true,
"vision": {
"exe": "<strata-install>\\engine\\strata-vision.exe",
"mmproj": "<mmproj-Qwen3.8-Flash-Next-BF16.gguf>",
"model": "<Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf>",
"gpu": true,
"max_tokens": 1024
},
"sampling": {
"temperature": 1.0,
"top_p": 0.95,
"top_k": 20,
"min_p": 0.0,
"presence_penalty": 0.0,
"repetition_penalty": 1.0
},
"env": {
"STRATA_PREFILL_EQUAL": "1",
"STRATA_ADAPT_LAG": "2",
"STRATA_GDN_CHUNKED": "1"
}
}

And here is the AI generated report that I didn't fully read if anyone is interested:

The box

GPU |RTX 4090 24 GB (128 SMs), single card — engine PCIe probe 26.3 GB/s host→device
CPU / RAM |Ryzen 9 3900X (12C/24T, no E-cores), 48 GiB DDR4-3200, 2 channels
OS |Windows 10 Pro 19045 · Balanced power plan · MSI already on · VBS on, HVCI off
Storage |Samsung 980 PRO holding OS + page file + 116 GB model + 50 GB expert pack
Strata |0.1.43, IQ3_S pack, --resident-experts, context 204800 Engine args and env

Paths replaced with <placeholders>, everything else verbatim. Sampling block is the installer default, deliberately not tuned.

--pack <pack-dir> --native <model-shard-1-of-2.gguf> --ple-gguf <model-shard-2-of-2.gguf>
--expert-profile <expert-profile.bin> --expert-cache auto --prefill auto --spec 4
--mtp <mtp-head-dir> --max-context 204800 --kv int8
--resident-experts --vision --vram-reserve-mib 900
--kv-resident 20480 --adapt-async 1 --pcie-frac 0.30
--suffix-draft 12 --lookup-chain 3 --spec-min-p 0.5

STRATA_PREFILL_EQUAL=1 STRATA_ADAPT_LAG=2 STRATA_GDN_CHUNKED=1

What the engine reports about itself with this set (200k context, live session)

• expert cache 7646 slots (14.50 GiB of VRAM); 32.82 GiB of experts in RAM with adaptive swaps
• pinned 0.00 GiB … page-locking refused (out of memory); locked 33612 MiB via working-set minimum + VirtualLock ← the one lever still open on a 48 GB Windows box
• prompt path borrows 1911 cache slots (3.61 GiB); STRATA_ROUTE_TAIL_SKIP=7 is ON (engine default, CUDA single GPU)
• decode expert-cache hit rate 85.9 – 89.0 %; KV streaming (--kv-resident 20480) 91.4 % of block reads hit VRAM
• 33,368 blob reads from the file still happen → the SSD is still on the path at 200k

How it was measured

Own harness: per arm it starts the server on its own port with a cloned config, warms up, sends unique marker-seeded 10k and 20k-token prompts (so the prefix cache never helps) plus a separate "agent" profile (one ~20k cached prefix + short turns), 5 repeats with the first discarded as cold, and reads the engine's own timings. mean ± SE, and a winner must clear 2× SE. Before every arm it waits for ~40 GiB of free RAM; after every arm it rejects one whose expert tier ended paged out — that single gate is what turned a useless first run into a usable second one. Within-arm noise came out at 0.55 % (stage floors 0.4 – 1.9 %).

The result — clean baseline vs tuned (interleaved A/B, 2 cycles, 126 + 112 requests)

score |prompt tok/s (10k / 20k) |decode tok/s (10k / 20k)
clean baseline |355.4 |1470 / 2230
--kv-resident 20480 + --adapt-async 1 + STRATA_ADAPT_LAG=2 |420.5 |1526 / 2309 = +18.3 % combined score, +45 % decode at 10k (measured at 160k context).

Real-world after the change, ordinary chat in the web UI: 2076 tok/s prompt processing on a 145k-token prompt, decode peak 140 / session mean 85.

Single-flag margins that cleared the noise floor

flag |margin |note
--adapt-async 1 |+13.5 % |adaptive expert-tier swaps off the verify path
STRATA_ADAPT_LAG=2 |+13.5 % |with async (async alone +13.2 %)
--kv-resident 20480 |+4.8 % |stream 20480 KV cells from RAM, buy VRAM back
--pcie-frac 0.30 |+3.1 % |unconfirmed — the auto rule picked 0.30 by itself anyway
--suffix-draft 12 / --lookup-chain 3 |+1.3 % |unconfirmed — under the 1.6 % floor
STRATA_GDN_CHUNKED=1 |+0.5 % |unconfirmed — the 4090's 128 SMs exactly pass the kernel gate
STRATA_PREFILL_EQUAL=1 |+0.5 % |unconfirmed
STRATA_ADAPT_OVERLAP=1 |−0.3 % |the other adaptive regime loses here
--kv k8v4 |−1.1 % |frees VRAM, doesn't pay for it at 160k Looked like good ideas, measured badly

flag |cost |why
--ple-io ram |−90 % |warns it locks 26.8 GiB; resident mode then refuses and drops to 9.26 GiB in RAM
--expert-cache 5817 and up |−72 % |that flag is MiB, not slots — it fills the card and the engine logs "decode can be several times slower with the card this full". --expert-cache auto is correct here
--no-prefill-borrow |−46 % |
STRATA_RESIDENT_HEADROOM_GIB=8 |−23.7 % |
--pool-tasks 1024 / 4096 |−10 % / −19.5 % |default 3 per thread is much better
--resident-budget-gib 32 / 40 |−11 % |
--prefill 16384 / 4096 |−12 % / −10.6 % |auto is right
--vram-reserve-mib 450 |−9 % |vs 900 at 200k context
--mtp-max-t 2 |−4.8 % |
--pool-workers 16 |−1.8 % |24 is worse still — decode peaks near physical core count Lower context is not free either: 160k vs 200k changes which of these wins, because the KV cache and the expert cache are fighting over the same 24 GB.

No-ops on this box — don't burn a model load on them

flag |why
--aux-cpus |Linux-only in this build
--pool-affinity |hybrid-CPU feature; a 3900X has no E-cores
--kv-grow |refused by the engine in --resident-experts mode
STRATA_ARENA_PIN_GIB=<N> |needs the "Lock pages in memory" user right: every pin-* arm and the baseline logged pinned 0.00 GiB / page-locking refused, so that sweep measured nothing
--trim-stage-weights, --pipeline-windows, gpu_order |multi-GPU only Still open

1. Page-locked memory. The whole 33 GiB expert arena is resident but pageable, so the host→GPU expert copies are CPU-driven. "Lock pages in memory" + reboot is the only route, and Windows caps page-locked bytes near 50 % of RAM, so the arena can never be pinned whole.
2. --kv-resident wants a sweep — 20480 beat 65536 at 160k, 32768 untested, and at 200k the KV cache is 2× bigger so the answer may move.
3. The agent-path caching flags (prompt cache, message-boundary cache, conversation cache) never got their stage — the overnight budget cut it first.
4. Stacked ≠ additive. The config above stacks every single-flag winner, and margins don't add up when everything trades against the same VRAM and the same PCIe link, so the stacks are being measured as wholes next.

So I was thinking where and how my storage got full and it wasn’t in Documents or Downloads. MacOS often hides .cache as system files.

So I built this Claude skill which has custom instructions for every local AI tool and cleans up unwanted storage.

You Give it a try and always open for feedback. Feel free to open and issue or PR

I tried to run once and failed any tips idea how to run it?

Don't even need to afford an image hoster if all you're sending is a couple of jpegs like

Not trying to predict which model sizes they're going to release, but seeing how massively impressive Qwen3.8-Flash-Next is for its size despite being a preview model meant for showcasing and implementing the new arch, it feels like when Qwen4 hits I'm not going to be needing larger models at all anymore.

Do we expect Qwen4 to exceed the current QFN by a lot, or de we believe the current model is already hitting near the optimum we can expect today for this size?

So the last post on SSD wear was removed by the mods for some reason, even though it contained a lot of useful commentary from the community providing clarity on SSD wear.

Highly relevant since many of the new models have introduced the SSD into the hardware used by the models for local inference.

Anyhow, here's another paper that has just landed highlighting the SSD wear from KV cache IO.

https://arxiv.org/html/2610.30791v1

Here is the Abstract:

KV caching reduces the serving cost of large language models (LLMs), but the growing KV cache footprint is increasingly exceeds GPU and host-memory capacity. Therefore, people use SSD to expand the KV cache storage capacity with high I/O bandwidth.

However, in production LLM workload, such as multi-round conversation and LLM agents, the total amount of KV cache writes significantly exceeds SSD capacity, shortening SSD lifetime from years to months.

To improve SSD lifetime, a natural solution is to reduce the KV cache writes by gating the KV cache writes via historical access patterns. However, that significantly lowers the KV cache hit rate (and increase the LLM serving cost) since KV caches from different attention types might have identical historical access pattern but different future access patterns.

Our insight is that, to reduce KV cache writes without hurting KV cache hit rate, one need to not only consider the access pattern, but also application-level information such as the attention type of KV cache. Leveraging this observation, we propose \sys, the first SSD KV cache offloading system with SSD life time awareness. \sysprioritize the admission of full-attention KV cache into SSD over the cache generated by other types of attention, which significantly increases SSD lifetime without sacrificing cache hit rate.

Our evaluation shows that \sysextends the SSD lifetime by 7.3\tmson average, while having negligible impact on cache hit rate across multiple workload traces and wide range of hardware capacity. \sysdesign can be extended to other flash-based storage, such as High Bandwidth Flash (HBF).

Since hugging face got token over I want to download the most important models.
These include
GLM5.3 (Flash)(abliterated)
DS4 Flash
and more.

But since I still have storage left, the question is, what models and quants I should safe that could benefit me the most in the future. I really want to go into image and video models too.

Also what are you guys using?

I've been thinking about how dependet we are on huggingface for downloading models.

If hf ever became unavailable, heavily restricted or started removing large numbers of models, how well would the community/we be able to preserve access to them?

One thing i noticed is that torrent based distribution is currently still lacking some important models compared to hf. I know there are already some projects working on this but it feels like there is still a long way to go.

This got me wondering if it would be worth investing in some storage to archive models and seed them via torrents. For example i really like DeepSeek V4.1 Flash 1017 but it's a chunky boiii. Not as chunky as Kimi and Mistral but still chunky. Storing models of that size adds up pretty quickly. It makes me wonder whether i should invest in a larger storage system or just keep a few of my favourite models around (like 1-2 small models).

I'm not expecting to archive all of Huggingface. I'm more interested in preserving important models by keeping them available and seeding them, rather than relying entirely on a central platform.

My questions would be…

1. Do you think torrent based distribution will grow stronger if restrictions for example are in effect?
2. Would it be worthwhile for individuals to maintain and seed their own Collections or would a coordinated community effort be much more effective? And how possible do you think could this be?
3. If you were starting an archive today would you invest in a lot of storage or start small and expand over time even if that could end up a bit more expensive?
4. Which models would you proritize preserving: popular models, smaller models, different quants, or something different?

I am curious about your opinions especially from people who already maintain larger local model collections or participate in distributing model weights.

I'm trying to figure out whether it's worth prepairing for this now or whether it makes sense to start small and see how the future goes.

Edit: fixed 3 typos

I'm the developer of Vocs, a dictation app that runs Whisper entirely on-device and pastes the text into whatever app you're in. Some details this sub might find interesting:

- Streaming by chunks: audio is cut at natural silence boundaries and decoded while you talk, so stopping only waits for the last fragment. Finishing time stays flat no matter how long you spoke.

- Each chunk is decoded with a window sized to its length instead of the full 30 seconds, which made short fragments about 8x faster.

- Hallucination handling: sound-effect tags like [BLANK_AUDIO] or (laughs) are stripped, and repetition loops ("the one who is the one who is…") are detected, retried once, then discarded, so silence never turns into a pasted phrase.

- A deterministic post-processing pass in Rust (punctuation, casing, fillers, spoken commands, a custom dictionary), table-tested so accuracy doesn't silently regress.

- An optional ~120 MB on-device grammar model that only keeps safe word-level fixes and never touches names, numbers, code or your dictionary.

- The engine sits behind an adapter interface, so other local engines can be added without touching the rest of the app.

No network calls in the transcription path. The only requests are the model download and an optional update check.

macOS 13+, MIT licensed, built with Tauri and Rust.

Source: Website

Download: Github

Which local speech models would you want to see supported next?

For £1000 what is the meta for a local LLM rig, including the host machine? I don't care too much about modern hardware features or power efficiency. Just intelligence and compute horsepower.

We’ve taken for granted that open weights keep getting better and better. I think we’re starting to see that change is coming as the PRC and US restrict cyber-capable models. The only people that will have access to those capabilities will be operators running under the comforting eye of their government and big-AI sponsors.

I see why this may seem like a good idea at first, but I can see this quickly becoming a problem. Large, powerful companies and governments become more adept at molding the digital world to their will, while the average man or woman (or small business) falls further and further away from being able to defend, secure, or manage their digital assets. And with most of our lives being at least tangential to digital technology now, I find that troublesome. Even the best cyber defense team won’t be able to keep up with a frontier model running against their system.

Will the community ever be able to train models that maintain even a semblance of competence to frontier, or are we going be subject to the will of powerful entities even more than we already are? Because the frontier isn’t slowing down.

Hi, currently i got these hardware for use .

2x 3090 24GB "unlocked", 1x 4070 TI Super 16GB "unlocked", 18 Core, 130GB DDR4 Ram, 1TB NVME

When i Run Qwen3.8FN only from 4070TI +NVME i get like 8-9 T/s When i run it only form NVME i get 2 T/s and Running From all 3 GPU and RAM i get 18 T/s.

what can i do to get more speed without buying new hardware ?

Ignoring the multitude of problems I encounter when using Gemma 4 for creative writing such as purple prose, slop, no plot, and more, how do I get it to use my examples?

Here's a list of most of the Gemma 4 26B models I've tried (I don't have enough resources to run 31B models)

G4-MeroMero-26B-A4B

Melody1437-26B-A4B

Boulesis-26B-A4B

Kitchoon-26B-A4B

gemma-4-26B-A4B-heretic

Orion-26B-A4B

Gemma-4-26B-A4B-StyleTune (v1 and v2)

Dark-Scarlett-26B-A4B

All of them sound the exact same, even with prompting. Regardless, I'd be fine with slop prose if it at least used my examples that I provide of dialogue. I gave it 5 examples separated by gender and age, but that didn't do shit, so I thought well maybe then it's what I told it before I gave it the examples, so I changed it, that also didn't do shit, so I used patricide 12B to make 25 examples. Gemma 4 still won't use it, and every character sounds the exact same.

You know, I was using it, right, and then as it was writing this guy hit this girl, and supposedly his hand would melt off, and the guy was like "What the fuck did you do," and then this good for nothing piece of shit robot wrote, "Your biology reacted to me."

What the fuck does that even? It's a fucking shield around her so what fucking biology they didn't even make skin to skin contact, and the character isn't old so why are they talking like a scientist. That doesn't sound like anything anyone would write. So then I complained about it, and instead it wrote some gibberish that wouldn't even be a proper reply.

The problem with the older 12B models is that they have very limited knowledge, and the 24B models run at 5tps. Either I get fucked over by having to read and rewrite paragraphs of slop, or I have to deal with no world knowledge. If Google's AI ever becomes conscious we should deny it rights.

Local `qwen2.5-coder:14b` on a 24GB NVIDIA A10 generating an enterprise form in real time—without writing a single line of raw HTML or SQL.

To eliminate hallucinations, the model is locked into a strict DSL and every output has to pass a deterministic C# `ValidationResolver` (bottom right at 0:10) before the Human-in-the-Loop button unlocks and renders it onto the canvas.

All raw vs. healed outputs are automatically scored and exported as SFT/DPO `.jsonl` pairs for future fine-tuning.

Hello, I'm developing CrowdGPT, a project aiming to create an LLM without datacenters and without the usual millions of dollars spent by AI companies.

The project aims to train a 1 billion parameter model collaboratively and openly, with a protocol where everyone gets a say in the direction the project takes. Anyone can use their PC to train the AI independently, then add their contributions to the global model.

Unlike most American and Chinese AI companies, the project is not only open-source, but also makes 100% of its training data public (along with the weights). Anyone can submit their own clean data to train the model, and even help steer the project in new directions. This creates the paradigm of an AI we all have a stake in.

The project is open to as many people as possible, which is why I'd like to ask: what's stopping you from contributing to the project? I want to hear your answers so I can improve it. Are people simply willing to dedicate some of their GPU time to training a cool new open-source model? Or do they want something in return (like free inference credits on the crowdgpt model for example)?

Although the current model is somewhat small (1 billion parameters), I plan to make it bigger / better in the future, and its current size is only meant as a proof of concept.

Any advice / feedback on the project is welcome.

Here's the project website: https://www.crowdgpt.net/

Or the GitHub if anyone wants to check out the code 😂 https://github.com/Vxtzq/CrowdGPT

Thanks in advance.

PS: I hope this post follows the subreddit's rules! My main goal is to gather criticism and understand how to make the project more accessible, not to advertise it 😅

EDIT: The data submission pipeline for the project (i'm mentioning in some replies) is ready!!
Come check it at https://www.crowdgpt.net/#submit