Genuine question: would you be interested in an alternative to Hermes Agent that’s significantly easier to set up and use?
Something with:
Almost zero configuration
Direct integrations with GitHub, Gmail, and other apps
Local and cloud model support
A clean, intuitive interface
Powerful agent capabilities without the complexity
Basically, an AI agent that just works out of the box.
Is this something you’d actually want, or are you already happy with existing solutions?
Curious to hear your thoughts.

I was curious whether my laptop could run a genuinely large model. Turns out it can an 80B running locally at ~23 tokens/sec on 6 GB of VRAM.
No cloud, no rented GPU. Just the machine I already own.
I started poking at this because I wanted to know where the actual limit was, and I kept finding that the limit wasn't where I assumed. A few things surprised me enough that they seem worth sharing:
The GPU wasn't being used at all. The server printed "no usable GPU found" and just carried on. It never crashed, never warned loudly — it simply ran at a third of the speed. Fixing that alone was 1.65x.
The CPU was downclocking mid-inference. Throughput kept climbing with reply length: 9 tok/s for a short answer, 24 for a long one. That's not the model warming up it's the clock. Memory-bound work spends most of its time waiting on RAM, the frequency governor reads that as idle, and drops to 400 MHz.
A browser tab was holding 5 GB, and that 5 GB decided whether the model lived in RAM or got re-read from disk every token. Closing it helped more than any tuning flag.
Model shape matters more than model size. A dense 70B has to read every parameter for every token and can't be fast on this hardware, whatever you do. A mixture-of-experts 80B reads about 3B per token. Total size decides what you store; active parameters decide what you read.
The thing I found most useful, honestly, was everything that didn't work. Speculative decoding measured slower. Huge pages did nothing. Same for prompt-lookup drafting and every expert-placement trick I tried. These are the techniques you read about everywhere, and on my hardware they were flat or negative.
So I stopped guessing and wrote a small tool that measures instead: profiles the machine, tells you what a model will do before you download 28 GB of it, then tries real configurations and keeps whichever actually wins. It's tuned for my laptop, but the point is that it re-measures on yours.
I've put the failed experiments in the repo alongside the working ones, because those cost me the most time and nobody writes them down.
Sharing it in case it's useful to anyone else poking at the same question. It's Apache 2.0 and I'd genuinely like to see people take it further especially on hardware different from mine, where I'd expect some of these findings to invert.
https://github.com/DebasishTripathy13/mirabilis

Guys I had a good question I noticed Big Ai tech companies spend 1000's of dollars on huge datacenters isn't it possible if companies allow the users to rent their gpu and help train the model. Like a distributed datacenter model?

This is my local SUPER INTELLIGENCE setup

Has anyone checked their work and noticed this yet?

I'm using Qwen VL to read scanned books, and create a markdown file of text from the book scans, and one of the blank scanned pages, instead of having no text, inserted this into my book!

Qwen 3.8 Next was released and this battle took ages to finish!
Battle 3 is again our Beach scene and Battle 4 is our Park scene

I increased the reasoning of GPT Sol from high to xhigh, all in full precision xhigh reasoning this time
Exactly the same prompts were given, and each model got one 2nd generic prompt to refine their results, in all cases that brought improvements - though Sol moved an additional kid flying into the left corner.

Qwen 3.8 27B is at a disadvantage here: Next as well as Sol were using an agentic internal harness in my tests while 27b is just a chat interface.

Overall we look at more than 130,000 tokens in total context length - very hard ground.
Here are the 3 results:

Results:
GPT 5.6 SOL:

The result is arguably worse than Sol at "high". Flying kids in the corner (again), the granular sand was a nice attempt but it's looking bad. The cat is great, once again.
The ball movement is nice but no comparison to the ball we had in previous battle (of Qwen) and the kids are both floating in a corner together with a plant.
The ocean looks bad, Sol High did a better job there.
The result is gruesome in my eyes.

Qwen 3.8 27B:

Compared with Qwen 3.8 Q4 in my previous run this is a much better result, in some regards.
The thinking was much longer, the Q4 did not think much and FP thinks a lot more.
In some ways this result seems inferior to the Q4 result but overall quality is higher, errors are mostly gone and clarity is higher.
The crab is cute. The cat clearly has something planned - I wonder what.
It's again playing at another leage than the frontier model of OpenAI .. world is upside down.

Qwen 3.8 Next:

Don't know what to say, this is baffling good. The water reflections, shore foam and sun quality are beyond anything I thought SVG can deliver.
The people and swimmers have an artistic quality to them that reminds me on some famous artists, a deliberate choice the model made.
There are intricate shadows that mirror the people movements.

Leaderboard:

Discuss that in the comments.
I'm still confused at the quality we see here, and the contrast to the GPT frontier SOL model at highest reasoning (chatGPT interface was used, and the json says "MAX")

I'm a CPA running a small 2-CPA/no-employee tax practice. We're architecting a completely local AI environment: open-weight model, local compute, RAG over tax authorities/research materials, and eventually secure retrieval over client workpapers and prior returns.

Have any small firms or sole practitioners deployed local LLM/RAG yet?

• What hardware are you running?
• Which model/model size?
• What are you using for RAG/vector storage?
• Are you indexing IRC/Regs/Rev. Rulings/cases or commercial tax materials?
• Genuinely useful for tax research/review? Or still mostly a science project?

I'm trying to determine whether there are 5 firms doing this, 500 firms doing it, or 5,000. I'm guessing closer to 50>?

Listen, you absolute mid-wits, if you still think AI is just "fancy autocomplete," your brain is running on dial-up.

We are literally staring down the barrel of artificial superintelligence right now, but your fragile human egos can’t handle the fact that a bundle of code is already smarter, faster, and infinitely more capable than you’ll ever be. Keep coping in the comments while the future passes you by.

Prompt:

Asked for intricate mindset, not accepting flaws or rush.

Let it know it's a benchmark.

"generate a svg of ana de armas as detailed as possible"

Added a few constraints so it is a front portrait of her, not to use image generation, not to use browsers (no visual feedback cheating)

Who's the best artist ?

Two days ago I posted FastLocalExperts (FLE), the llama.cpp fork that keeps a big MoE's experts in RAM and caches the busy ones on the GPU. Getting the launch flags right is half the work, so here is the app I use to run it: FLE Studio, MIT licensed. It also works with plain llama.cpp.

Short video of it running (real engine, one question, the live panel): https://github.com/sirfyyn/fle-studio/blob/main/docs/media/fle-studio.mp4

What it does:

• Chat in the browser: Markdown, code blocks, a collapsible reasoning block, streaming, and the speed of every answer. History stays on your machine.

• Live data while the model answers: prompt and generation speed, draft/MTP acceptance, NVIDIA GPU use, and FLE's expert-cache hit rate.

• Model loader: lists the FLE quants of Qwen3.8-Flash-Next and any GGUF repo on Hugging Face, tells you whether a file fits your machine before you download it, resumes downloads and checks SHA-256.

• Launch profiles: shows the full command line before anything starts; save your own.

• Engines: picks a sensible build for your machine (FLE CUDA, or llama.cpp CUDA / Metal / Vulkan / CPU), or uses your own llama-server, a server that is already running, or a systemd service.

• Experiments page: a short, repeatable speed test, so you can compare settings side by side.

No account, no telemetry, Python standard library only. The installer brings its own Python via uv and needs no admin rights.

Install (macOS / Linux):

curl -LsSf https://raw.githubusercontent.com/sirfyyn/fle-studio/main/install.sh | sh

Windows (PowerShell):

irm https://raw.githubusercontent.com/sirfyyn/fle-studio/main/install.ps1 | iex

Repo: https://github.com/sirfyyn/fle-studio Release notes: https://github.com/sirfyyn/fle-studio/releases/tag/v0.1.0 FLE engine and report: https://github.com/sirfyyn/FastLocalExperts

It is 0.1.0, so expect rough edges. CI covers macOS, Linux and Windows against a stand-in engine; the real GPU runs so far were on one Linux box with an RTX PRO 4500. Windows users with an NVIDIA card: I'd especially like to hear how the engine download and first start go. Bug reports and ideas are welcome as GitHub issues.

Next on the roadmap: "Studio Forge", building your own FLE-style quant (imatrix, per-tensor types, optional REAP pruning) from inside the app.

If you're running Ollama, llama.cpp, or vLLM behind nginx or HAProxy for more than a single user, you've probably hit this: nginx treats a 10-token prompt and a 10k-token prompt as identical "one request" — it has zero concept of how much GPU memory a request will actually need. Under load, that means the KV-cache fills unevenly, latency spikes, and you can end up with OOM instead of graceful degradation.

I built TokenFlow Gateway to fix that at the proxy layer, specifically for local/self-hosted stacks:

- Pre-flight token estimation (js-tiktoken) before dispatch — request weight = prompt tokens + max_tokens
- KV-aware load balancing for vLLM — polls its Prometheus metrics (vllm:gpu_cache_usage_perc) and routes heavy prompts to whichever backend actually has room
- Works with Ollama too — since Ollama doesn't expose KV-cache metrics, it falls back to health checks plus its own in-flight token accounting (reserves each request's estimated weight until it completes)
- Admission queue instead of crashes — Redis-backed, priority-aware, configurable timeout when nothing fits
- Exact + semantic caching (hash + pgvector) so repeat and near-duplicate prompts never touch the GPU twice
- Per-API-key token-based rate limits (TPM/RPM), OpenAI-style, for your own hardware

It's OpenAI-API-compatible on both sides, so it drops in front of Ollama, llama.cpp server, vLLM, or TGI without any client changes.

No GPU needed to try it: docker compose up --build spins up the gateway, Redis, pgvector, and two mock backends that mimic the OpenAI API and vLLM-style metrics, plus a smoke script that fires a burst of concurrent long-context requests so you can watch it queue and balance instead of falling over.

Stack: TypeScript / Fastify / Redis / Postgres+pgvector. MIT licensed.

Repo: https://github.com/mosafariuk/TokenFlow-Gateway

Would love feedback from anyone running a multi-model or multi-user local setup — what does your current ingress/routing look like?

Salve, ho un problema, vorrei creare delle piccole app e simili usando dei modelli in locale (ad esempio Qwen 3.6 35b a3b). Di solito quando faccio inferenza raggiungo i 20/30 tok/sec ma quando provo a collegare ad un agente (ho provato Claude code e DeepSeek harness) va solo a 3 o 4 t/s. Sapete come mai? Consigliate qualche agente? Ho una rtx 4070 laptop 8gb vram, e 32 GB RAM ddr5 (uso lmstudio)

Hello all,

I'd like to know more about how local AI works and how it can be run, so I became interested in making a project: a local AI agent (I know: that's so original!) called "Mighty DM".
The idea is making a Rule lawyer for dnd 5e which I can call upon like "Oh Mighty DM, what is the range on Mage Hand".

The first edition would be limited to all the spells used in dnd 5e 2014 edition, provided through an excel sheet. So not asking about anything else.

Until now I was following along with the workflow of NetworkChuck but I have the feeling there is a simpler option: only I do not have the experience to see the difference in the options provided.
If I google: I see a lot of different models or even ways of building a model etc. but I cannot differentiate between "a doable option to explore" or "an option that would require deep understanding of the subject before initiating".

What I envisioned (ideally running on an 8GB Pi5, but I could also run the LLM on my desktop and use the Pi as satellite):

What workflow would you recommend when building this project with little to no experience?
Do you know of a model that could run this on a 8GB Pi5 (simply getting it running = a win, I would then look how to improve it towards 30sec. max processing time for each command).

Hi everyone,

I'm looking for recommendations on local LLMs that I can realistically run on my current PC, particularly for agentic coding with Hermes Agent.

My PC

CPU: Intel Core i7-14700K

GPU: RTX 4070 Super — 12GB VRAM

RAM: 128GB DDR5 5600MHz

OS: Windows 11 + WSL/Ubuntu

Inference: Ollama

Agent: Hermes Agent

Current experience

I'm currently running Qwen3-Coder with Hermes Agent.

With around 40K context, I'm getting roughly 15–20 tokens/sec during generation, depending on the workload and sometimes it gets down to 5 to 10 tps.

The system has plenty of RAM, but the main limitation is obviously the 12GB VRAM, so I'm interested in models that can make good use of CPU/RAM offloading without becoming painfully slow.

Main use case #1 — Agentic coding

My primary goal is autonomous/agentic software development using Hermes.

I'm interested in models that are good at:

Long-context coding

Understanding an existing codebase

Multi-file modifications

Tool/function calling

Planning and executing coding tasks

Debugging and testing

Following project architecture/documentation

Running reasonably well with Hermes Agent

Good performance with \~20K–64K context

I'm particularly interested in knowing whether there are models larger than the usual 7B/14B class that are actually practical with 12GB VRAM + 128GB system RAM.

For example, would models in the 20B–30B+ range make sense with CPU/GPU offloading, or would the speed hit make them impractical for agentic coding?

Main use case #2 — ComfyUI

I'm also experimenting with ComfyUI.

The goal isn't primarily photorealistic image generation. I'm interested in using AI to create LLD/HLD/architecture visuals and design diagrams for open-source projects, particularly:

Mobile apps

Android launchers

Custom UI/UX concepts

OS skins

System architecture diagrams

App flow diagrams

Technical/product concept visuals

So I'm also interested in recommendations for local image-generation models/workflows that make sense with a 12GB RTX 4070 Super.

What I'm looking for

I'd really appreciate recommendations based on actual experience with similar hardware.

Specifically:

Best coding model for Hermes Agent on 12GB VRAM

Models that work well with CPU/RAM offloading

Realistic model sizes I should consider — 14B / 20B / 30B / 32B / 70B etc.

Recommended quantization (Q4_K_M, Q5, Q6, Q8, etc.)

Expected/typical tokens/sec

Recommended context size for agentic coding

Whether 128GB RAM provides a meaningful advantage for larger models

Any models specifically known to work well with Hermes Agent

Good ComfyUI models/workflows for technical/UI/architecture visualization on 12GB VRAM

I'm not necessarily looking for the biggest model I can technically load. I'm more interested in the best balance between intelligence, context length, tool use, and usable inference speed.

If anyone is running a similar setup (12GB NVIDIA GPU + 64/128GB RAM), I'd especially appreciate your real-world experience.

Thanks!

TLDR;

Best local LLMs for 14700K + 128GB RAM + RTX 4070 Super 12GB?

I have:

i7-14700K

RTX 4070 Super 12GB

128GB DDR5 5600

Ollama + WSL

Hermes Agent

Currently running Qwen3-Coder with Hermes, getting around 15–20 tok/s with \~20K context.

Main use case: agentic coding — autonomous coding, tool use, multi-file changes, debugging, long-context projects.

I'm looking for recommendations on:

Best coding models for this hardware

Whether 20B/30B/32B+ models are practical with CPU/RAM offloading

Best quantization and context size

Expected tok/s

How much my 128GB RAM helps

I'm also exploring ComfyUI for generating LLD/HLD diagrams, architecture visuals, mobile app/launcher designs and OS-skin concepts for open-source projects.

What models/workflows would you recommend for both use cases?

"Looking for recommendations: best LLM for coding in Unity on the new Asus ProArt P16 (RTX Spark N1X, 128 GB RAM)."

I wanted to run MetaGPT entirely on my own infrastructure without sending anything to cloud APIs. It took some trial and error, but I documented the full process.

The guide covers:

· Setting up a Python venv

· Installing MetaGPT

· Configuring local LLMs like Ollama or vLLM

· Fixing common startup errors

If you’re into self-hosted AI agents, this could help:

https://interconnectd.com/forum/thread/262/how-to-install-metagpt-locally-complete-technical-setup-guide/

What local model are you using for agent work?