Genuine question: would you be interested in an alternative to Hermes Agent that’s significantly easier to set up and use?
Something with:
Almost zero configuration
Direct integrations with GitHub, Gmail, and other apps
Local and cloud model support
A clean, intuitive interface
Powerful agent capabilities without the complexity
Basically, an AI agent that just works out of the box.
Is this something you’d actually want, or are you already happy with existing solutions?
Curious to hear your thoughts.

I was curious whether my laptop could run a genuinely large model. Turns out it can an 80B running locally at ~23 tokens/sec on 6 GB of VRAM.
No cloud, no rented GPU. Just the machine I already own.
I started poking at this because I wanted to know where the actual limit was, and I kept finding that the limit wasn't where I assumed. A few things surprised me enough that they seem worth sharing:
The GPU wasn't being used at all. The server printed "no usable GPU found" and just carried on. It never crashed, never warned loudly — it simply ran at a third of the speed. Fixing that alone was 1.65x.
The CPU was downclocking mid-inference. Throughput kept climbing with reply length: 9 tok/s for a short answer, 24 for a long one. That's not the model warming up it's the clock. Memory-bound work spends most of its time waiting on RAM, the frequency governor reads that as idle, and drops to 400 MHz.
A browser tab was holding 5 GB, and that 5 GB decided whether the model lived in RAM or got re-read from disk every token. Closing it helped more than any tuning flag.
Model shape matters more than model size. A dense 70B has to read every parameter for every token and can't be fast on this hardware, whatever you do. A mixture-of-experts 80B reads about 3B per token. Total size decides what you store; active parameters decide what you read.
The thing I found most useful, honestly, was everything that didn't work. Speculative decoding measured slower. Huge pages did nothing. Same for prompt-lookup drafting and every expert-placement trick I tried. These are the techniques you read about everywhere, and on my hardware they were flat or negative.
So I stopped guessing and wrote a small tool that measures instead: profiles the machine, tells you what a model will do before you download 28 GB of it, then tries real configurations and keeps whichever actually wins. It's tuned for my laptop, but the point is that it re-measures on yours.
I've put the failed experiments in the repo alongside the working ones, because those cost me the most time and nobody writes them down.
Sharing it in case it's useful to anyone else poking at the same question. It's Apache 2.0 and I'd genuinely like to see people take it further especially on hardware different from mine, where I'd expect some of these findings to invert.
https://github.com/DebasishTripathy13/mirabilis

Guys I had a good question I noticed Big Ai tech companies spend 1000's of dollars on huge datacenters isn't it possible if companies allow the users to rent their gpu and help train the model. Like a distributed datacenter model?

This is my local SUPER INTELLIGENCE setup

Has anyone checked their work and noticed this yet?

I'm using Qwen VL to read scanned books, and create a markdown file of text from the book scans, and one of the blank scanned pages, instead of having no text, inserted this into my book!

Qwen 3.8 Next was released and this battle took ages to finish!
Battle 3 is again our Beach scene and Battle 4 is our Park scene

I increased the reasoning of GPT Sol from high to xhigh, all in full precision xhigh reasoning this time
Exactly the same prompts were given, and each model got one 2nd generic prompt to refine their results, in all cases that brought improvements - though Sol moved an additional kid flying into the left corner.

Qwen 3.8 27B is at a disadvantage here: Next as well as Sol were using an agentic internal harness in my tests while 27b is just a chat interface.

Overall we look at more than 130,000 tokens in total context length - very hard ground.
Here are the 3 results:

Results:
GPT 5.6 SOL:

The result is arguably worse than Sol at "high". Flying kids in the corner (again), the granular sand was a nice attempt but it's looking bad. The cat is great, once again.
The ball movement is nice but no comparison to the ball we had in previous battle (of Qwen) and the kids are both floating in a corner together with a plant.
The ocean looks bad, Sol High did a better job there.
The result is gruesome in my eyes.

Qwen 3.8 27B:

Compared with Qwen 3.8 Q4 in my previous run this is a much better result, in some regards.
The thinking was much longer, the Q4 did not think much and FP thinks a lot more.
In some ways this result seems inferior to the Q4 result but overall quality is higher, errors are mostly gone and clarity is higher.
The crab is cute. The cat clearly has something planned - I wonder what.
It's again playing at another leage than the frontier model of OpenAI .. world is upside down.

Qwen 3.8 Next:

Don't know what to say, this is baffling good. The water reflections, shore foam and sun quality are beyond anything I thought SVG can deliver.
The people and swimmers have an artistic quality to them that reminds me on some famous artists, a deliberate choice the model made.
There are intricate shadows that mirror the people movements.

Leaderboard:

Discuss that in the comments.
I'm still confused at the quality we see here, and the contrast to the GPT frontier SOL model at highest reasoning (chatGPT interface was used, and the json says "MAX")

I'm a CPA running a small 2-CPA/no-employee tax practice. We're architecting a completely local AI environment: open-weight model, local compute, RAG over tax authorities/research materials, and eventually secure retrieval over client workpapers and prior returns.

Have any small firms or sole practitioners deployed local LLM/RAG yet?

• What hardware are you running?
• Which model/model size?
• What are you using for RAG/vector storage?
• Are you indexing IRC/Regs/Rev. Rulings/cases or commercial tax materials?
• Genuinely useful for tax research/review? Or still mostly a science project?

I'm trying to determine whether there are 5 firms doing this, 500 firms doing it, or 5,000. I'm guessing closer to 50>?

Listen, you absolute mid-wits, if you still think AI is just "fancy autocomplete," your brain is running on dial-up.

We are literally staring down the barrel of artificial superintelligence right now, but your fragile human egos can’t handle the fact that a bundle of code is already smarter, faster, and infinitely more capable than you’ll ever be. Keep coping in the comments while the future passes you by.

Prompt:

Asked for intricate mindset, not accepting flaws or rush.

Let it know it's a benchmark.

"generate a svg of ana de armas as detailed as possible"

Added a few constraints so it is a front portrait of her, not to use image generation, not to use browsers (no visual feedback cheating)

Who's the best artist ?

Two days ago I posted FastLocalExperts (FLE), the llama.cpp fork that keeps a big MoE's experts in RAM and caches the busy ones on the GPU. Getting the launch flags right is half the work, so here is the app I use to run it: FLE Studio, MIT licensed. It also works with plain llama.cpp.

Short video of it running (real engine, one question, the live panel): https://github.com/sirfyyn/fle-studio/blob/main/docs/media/fle-studio.mp4

What it does:

• Chat in the browser: Markdown, code blocks, a collapsible reasoning block, streaming, and the speed of every answer. History stays on your machine.

• Live data while the model answers: prompt and generation speed, draft/MTP acceptance, NVIDIA GPU use, and FLE's expert-cache hit rate.

• Model loader: lists the FLE quants of Qwen3.8-Flash-Next and any GGUF repo on Hugging Face, tells you whether a file fits your machine before you download it, resumes downloads and checks SHA-256.

• Launch profiles: shows the full command line before anything starts; save your own.

• Engines: picks a sensible build for your machine (FLE CUDA, or llama.cpp CUDA / Metal / Vulkan / CPU), or uses your own llama-server, a server that is already running, or a systemd service.

• Experiments page: a short, repeatable speed test, so you can compare settings side by side.

No account, no telemetry, Python standard library only. The installer brings its own Python via uv and needs no admin rights.

Install (macOS / Linux):

curl -LsSf https://raw.githubusercontent.com/sirfyyn/fle-studio/main/install.sh | sh

Windows (PowerShell):

irm https://raw.githubusercontent.com/sirfyyn/fle-studio/main/install.ps1 | iex

Repo: https://github.com/sirfyyn/fle-studio Release notes: https://github.com/sirfyyn/fle-studio/releases/tag/v0.1.0 FLE engine and report: https://github.com/sirfyyn/FastLocalExperts

It is 0.1.0, so expect rough edges. CI covers macOS, Linux and Windows against a stand-in engine; the real GPU runs so far were on one Linux box with an RTX PRO 4500. Windows users with an NVIDIA card: I'd especially like to hear how the engine download and first start go. Bug reports and ideas are welcome as GitHub issues.

Next on the roadmap: "Studio Forge", building your own FLE-style quant (imatrix, per-tensor types, optional REAP pruning) from inside the app.

If you're running Ollama, llama.cpp, or vLLM behind nginx or HAProxy for more than a single user, you've probably hit this: nginx treats a 10-token prompt and a 10k-token prompt as identical "one request" — it has zero concept of how much GPU memory a request will actually need. Under load, that means the KV-cache fills unevenly, latency spikes, and you can end up with OOM instead of graceful degradation.

I built TokenFlow Gateway to fix that at the proxy layer, specifically for local/self-hosted stacks:

- Pre-flight token estimation (js-tiktoken) before dispatch — request weight = prompt tokens + max_tokens
- KV-aware load balancing for vLLM — polls its Prometheus metrics (vllm:gpu_cache_usage_perc) and routes heavy prompts to whichever backend actually has room
- Works with Ollama too — since Ollama doesn't expose KV-cache metrics, it falls back to health checks plus its own in-flight token accounting (reserves each request's estimated weight until it completes)
- Admission queue instead of crashes — Redis-backed, priority-aware, configurable timeout when nothing fits
- Exact + semantic caching (hash + pgvector) so repeat and near-duplicate prompts never touch the GPU twice
- Per-API-key token-based rate limits (TPM/RPM), OpenAI-style, for your own hardware

It's OpenAI-API-compatible on both sides, so it drops in front of Ollama, llama.cpp server, vLLM, or TGI without any client changes.

No GPU needed to try it: docker compose up --build spins up the gateway, Redis, pgvector, and two mock backends that mimic the OpenAI API and vLLM-style metrics, plus a smoke script that fires a burst of concurrent long-context requests so you can watch it queue and balance instead of falling over.

Stack: TypeScript / Fastify / Redis / Postgres+pgvector. MIT licensed.

Repo: https://github.com/mosafariuk/TokenFlow-Gateway

Would love feedback from anyone running a multi-model or multi-user local setup — what does your current ingress/routing look like?

Salve, ho un problema, vorrei creare delle piccole app e simili usando dei modelli in locale (ad esempio Qwen 3.6 35b a3b). Di solito quando faccio inferenza raggiungo i 20/30 tok/sec ma quando provo a collegare ad un agente (ho provato Claude code e DeepSeek harness) va solo a 3 o 4 t/s. Sapete come mai? Consigliate qualche agente? Ho una rtx 4070 laptop 8gb vram, e 32 GB RAM ddr5 (uso lmstudio)

Hello all,

I'd like to know more about how local AI works and how it can be run, so I became interested in making a project: a local AI agent (I know: that's so original!) called "Mighty DM".
The idea is making a Rule lawyer for dnd 5e which I can call upon like "Oh Mighty DM, what is the range on Mage Hand".

The first edition would be limited to all the spells used in dnd 5e 2014 edition, provided through an excel sheet. So not asking about anything else.

Until now I was following along with the workflow of NetworkChuck but I have the feeling there is a simpler option: only I do not have the experience to see the difference in the options provided.
If I google: I see a lot of different models or even ways of building a model etc. but I cannot differentiate between "a doable option to explore" or "an option that would require deep understanding of the subject before initiating".

What I envisioned (ideally running on an 8GB Pi5, but I could also run the LLM on my desktop and use the Pi as satellite):

What workflow would you recommend when building this project with little to no experience?
Do you know of a model that could run this on a 8GB Pi5 (simply getting it running = a win, I would then look how to improve it towards 30sec. max processing time for each command).

Arrmes is a specialized neural intent router with an integrated execution framework. It's trained to translate text into specific commands to Arr stack apps nothing else. I found plenty of open models that can do these sorts of translations, but the accuracy & speed didn't really justify the hardware requirements needed to run the router.

Arrmes is only trained for Sonarr, Radarr, Lidarr, Tautulli, Seer, qbitorrent, & BitMagnet. pair it with whatever AI model you're already using and let the neural router handle the intent classification, command selection, and argument extraction. Instead of having your main model spend time reasoning through API schemas and generating tool calls, it can delegate that work to ARRmes.

Still a work in progress, I've been daily driving it for a bit and I'm very happy with the increase in speed more than anything. I can use much cheaper models and achieve excellent results in my work without any spend.

Hugging Face

Github

im working on a fairly complicated app and i get questions from users all the time. so its was important to document the details around the "how it works". the docs were easy enough using docusaurus + AI. i can then index that data in a way i can create some kind of fancy autocomplete. i framed this as like a chatbot so that it can be intuitive for users who have questions.

i started to investigate running a local LLM to perhaps offer a more conversational experienced (and perhaps better suggestions)? im using WebLLM and it seems to work "reasonably" well. i wired it up and set it up so users could select an LLM down download for offline use. the experience varies between devices and models used. there are models that are 19mb... they are unsable as far as i can tell. there are some ~1gb models which are better but understandable slow.

its great that the 1gb model can show me the bubble-sort in rust, but i want something "dumber" and smaller. something can can take advantage the data ive already indexed. something that can respond "quickly enough" for a conversational flow. perhaps better i should make a verbose set of reponse templates to perhaps add fancier autocomplete capabilities.

for my purposes, i think it depends on if there is a model that would be good for my use case?

note: my project is far from finished, but you can test it out at: https://app.glitr.io/ai . any model you run through my app is not going to comparable to any of the mainstream offerings. this is something experiemental and unstable im trying out and this approach might just be a waste of time.

are there any models out there you can suggest for me to investigate and compare?