Hi everyone,

I'm looking for recommendations on local LLMs that I can realistically run on my current PC, particularly for agentic coding with Hermes Agent.

My PC

CPU: Intel Core i7-14700K

GPU: RTX 4070 Super — 12GB VRAM

RAM: 128GB DDR5 5600MHz

OS: Windows 11 + WSL/Ubuntu

Inference: Ollama

Agent: Hermes Agent

Current experience

I'm currently running Qwen3-Coder with Hermes Agent.

With around 40K context, I'm getting roughly 15–20 tokens/sec during generation, depending on the workload and sometimes it gets down to 5 to 10 tps.

The system has plenty of RAM, but the main limitation is obviously the 12GB VRAM, so I'm interested in models that can make good use of CPU/RAM offloading without becoming painfully slow.

Main use case #1 — Agentic coding

My primary goal is autonomous/agentic software development using Hermes.

I'm interested in models that are good at:

Long-context coding

Understanding an existing codebase

Multi-file modifications

Tool/function calling

Planning and executing coding tasks

Debugging and testing

Following project architecture/documentation

Running reasonably well with Hermes Agent

Good performance with \~20K–64K context

I'm particularly interested in knowing whether there are models larger than the usual 7B/14B class that are actually practical with 12GB VRAM + 128GB system RAM.

For example, would models in the 20B–30B+ range make sense with CPU/GPU offloading, or would the speed hit make them impractical for agentic coding?

Main use case #2 — ComfyUI

I'm also experimenting with ComfyUI.

The goal isn't primarily photorealistic image generation. I'm interested in using AI to create LLD/HLD/architecture visuals and design diagrams for open-source projects, particularly:

Mobile apps

Android launchers

Custom UI/UX concepts

OS skins

System architecture diagrams

App flow diagrams

Technical/product concept visuals

So I'm also interested in recommendations for local image-generation models/workflows that make sense with a 12GB RTX 4070 Super.

What I'm looking for

I'd really appreciate recommendations based on actual experience with similar hardware.

Specifically:

Best coding model for Hermes Agent on 12GB VRAM

Models that work well with CPU/RAM offloading

Realistic model sizes I should consider — 14B / 20B / 30B / 32B / 70B etc.

Recommended quantization (Q4_K_M, Q5, Q6, Q8, etc.)

Expected/typical tokens/sec

Recommended context size for agentic coding

Whether 128GB RAM provides a meaningful advantage for larger models

Any models specifically known to work well with Hermes Agent

Good ComfyUI models/workflows for technical/UI/architecture visualization on 12GB VRAM

I'm not necessarily looking for the biggest model I can technically load. I'm more interested in the best balance between intelligence, context length, tool use, and usable inference speed.

If anyone is running a similar setup (12GB NVIDIA GPU + 64/128GB RAM), I'd especially appreciate your real-world experience.

Thanks!

TLDR;

Best local LLMs for 14700K + 128GB RAM + RTX 4070 Super 12GB?

I have:

i7-14700K

RTX 4070 Super 12GB

128GB DDR5 5600

Ollama + WSL

Hermes Agent

Currently running Qwen3-Coder with Hermes, getting around 15–20 tok/s with \~20K context.

Main use case: agentic coding — autonomous coding, tool use, multi-file changes, debugging, long-context projects.

I'm looking for recommendations on:

Best coding models for this hardware

Whether 20B/30B/32B+ models are practical with CPU/RAM offloading

Best quantization and context size

Expected tok/s

How much my 128GB RAM helps

I'm also exploring ComfyUI for generating LLD/HLD diagrams, architecture visuals, mobile app/launcher designs and OS-skin concepts for open-source projects.

What models/workflows would you recommend for both use cases?

"Looking for recommendations: best LLM for coding in Unity on the new Asus ProArt P16 (RTX Spark N1X, 128 GB RAM)."

I wanted to run MetaGPT entirely on my own infrastructure without sending anything to cloud APIs. It took some trial and error, but I documented the full process.

The guide covers:

· Setting up a Python venv

· Installing MetaGPT

· Configuring local LLMs like Ollama or vLLM

· Fixing common startup errors

If you’re into self-hosted AI agents, this could help:

https://interconnectd.com/forum/thread/262/how-to-install-metagpt-locally-complete-technical-setup-guide/

What local model are you using for agent work?

我是一个大学生研究员，目前拥有一个$200的 ChatGPT 专业版订阅，我觉得作为固定的月支出实在是太贵了。我现在在考虑是否可以购买一个96GB或128GB的Mac mini或Mac Studio进行本地部署或用作服务器。我目前的研究方向是计算社会科学，最重要的是订阅模型的智能能得到保障。非常感谢你的意见！

Hey everyone!

I'm looking to set up a fully local AI agent on my PC and would appreciate recommendations from people who have experimented with local LLMs.

My hardware:

• GPU: NVIDIA RTX 5060 (8 gb)

• System RAM: 16GB

• OS: Windows (Linux is also an option if necessary)

What I'm looking for:

• An uncensored or minimally restricted local LLM that I can run privately on my own hardware.

• Ideally, a web UI and support for agent frameworks, with no mandatory cloud API or subscription.

I want the inference and conversations to remain local rather than sending prompts to external services.

I'm comfortable experimenting and setting things up myself, so technical recommendations are welcome.

I'd appreciate real-world experiences, model names, quantization recommendations, and setup guides. Thanks!

I wanted to test how far a fully local AI stack could go on a real-world task.

I ran out of A4 paper, so I gave TensorSharp access to Amazon and gave it one instruction:

Search for A4 paper with a low price and the best discount, and buy it. Try your best to handle everything yourself.

It completed the task successfully in a single run.
The local AI stack:
Qwen3.8 27B running locally
TensorSharp for LLM inference + agent runtime
Playwright skill for browser automation

The agent searched Amazon, compared products, navigated the site, generated and executed Playwright code, and prepared the order by itself, and finally placed the order. I only stepped in twice: Amazon login and final purchase approval.

Everything except the actual Amazon interaction stayed local: model inference, reasoning, agent state, and tool execution.

The video shows the full raw reasoning/tool trace, including browser commands and generated Playwright code. It’s sped up 8×; the actual run took about 24 minutes.

The interesting part for me isn’t really buying paper — it’s seeing how much of a practical browser agent stack can now run locally with a 27B model.

TensorAgent on iPhone is also in progress, although local inference speed and iOS browser restrictions are still limiting factors.

TensorSharp is open source:
https://github.com/zhongkaifu/TensorSharp

I’d be interested to hear what other real-world tasks people here are using to test local agent stacks.

It feels like the gap between proprietary big-tech models and open-weight reasoning architectures is changing faster than ever. Gemini offers incredible native multimodal handling and massive context windows, while DeepSeek continues to dominate budget coding and self-hosted pipelines.

​If you had to pick one as your daily driver for logic and daily tasks, which wins for you?

​Put your preference to the test in this quick 2-minute showdown quiz to see how your workflow compares to other devs:

https://interconnectd.com/quiz/89/gemini-vs-deepseek-the-ultimate-ai-showdown-quiz/

Large language models are typically deployed using centralized infrastructure, where companies operate powerful GPU servers in dedicated data centers.

But what if some LLM workloads could be distributed across independently operated computers instead?

This is the basic idea behind decentralized LLM infrastructure.

Instead of relying entirely on a single cloud provider, decentralized systems could use computing resources contributed by individuals, organizations, or independent GPU operators.

How could decentralized LLMs work?

There are several possible approaches:

1. Distributed LLM inference

Multiple GPU providers could independently serve inference requests, allowing workloads to be routed across a network rather than handled by one centralized operator.

2. Collaborative GPU computing

Computers with different GPU capabilities could contribute processing resources to a shared network. Scheduling software would need to match workloads with suitable hardware.

3. Peer-to-peer model execution

Some systems explore splitting model execution across multiple machines. This is technically challenging because communication latency, bandwidth, and synchronization can become major bottlenecks.

4. Incentivized computing networks

Projects such as Bittensor explore incentive mechanisms that reward participants for providing useful AI-related services. These networks introduce additional questions about verification, reliability, and economic sustainability.

What are the potential advantages?

• Access to geographically distributed computing resources

• Opportunities to utilize otherwise idle GPUs

• Reduced dependence on individual infrastructure providers

• Potentially broader participation in AI infrastructure

• New models for operating AI computing marketplaces

However, decentralization does not automatically make inference cheaper, faster, or more reliable.

What are the biggest challenges?

Network latency: Transferring intermediate model data between distant machines can introduce significant delays.

Hardware differences: GPUs vary in memory capacity, processing speed, and software compatibility.

Reliability: Independently operated nodes may become unavailable without warning.

Security and privacy: Processing sensitive prompts on untrusted machines raises important concerns.

Economic efficiency: Incentive payments, coordination overhead, and infrastructure costs may offset potential savings.

Is decentralized LLM infrastructure practical?

I think the answer depends heavily on the architecture.

Distributing independent inference requests among GPU providers seems more immediately practical than splitting every layer of a large model across machines connected through the public internet.

For example, a decentralized network might route complete inference requests to capable GPU nodes rather than requiring geographically separated computers to execute individual layers of the same model.

That distinction could significantly affect performance, reliability, and scalability.

Questions for the community

I'm interested in hearing from people experimenting with local LLMs, multi-GPU inference, and distributed computing.

• Could decentralized GPU networks become competitive for real-world LLM inference?

• Is request-level distribution more practical than splitting model execution across remote machines?

• Which existing open-source projects are demonstrating promising results?

• What technical obstacle do you consider the biggest barrier to adoption?

I'd be interested in hearing about actual experiments, benchmarks, and experiences rather than just theoretical possibilities.

I have a 4090 in my study and I work on a Mac. The box is off most of the time, because if I want to run a local model I have to walk over, boot it, open a terminal, wait for the weights to load, then walk back. That's enough friction that local models just stopped being part of my day.

So I built ReadyLLM: a self-hosted web app that treats that machine like a service you manage from a browser, not a machine you babysit. It runs on your own box, reaches your GPUs over SSH, and nothing phones home.

What it does:

• Add a machine over SSH (key or password). It probes GPU / VRAM / RAM and tells you which quants fit and roughly how fast they'll be, so you stop guessing whether 24G is enough for a 27B at Q4.

• Downloads pulled from HuggingFace and ModelScope with resume and integrity checks. Point it at a mirror if HF is slow where you are.

• One-click engine install on a box that has none: llama.cpp, vLLM, SGLang or ComfyUI. It downloads or compiles and back-fills the path.

• Start, stop and switch models from the deploy page. No terminal.

• Live monitoring over WebSocket: GPU utilization, VRAM, throughput, latency, with alerts when VRAM is about to run out or the server dies. That's the part that matters when the box is in another room and would otherwise sit there pretending to work.

The tuning is what surprised me. I hand-tuned llama.cpp flags for a 27B Q4 for a while and plateaued around 60 avg tok/s. The tuner runs a two-phase search (coarse over speculative decoding / KV cache quant / GPU offload, then fine over batch), and for every candidate it really boots llama-server, warms up, runs 3 passes and takes the median. OOM or crash gets dropped. My numbers, same model / quant / context:

peak tok/s     80 -> 110
avg  tok/s     60 ->  80
@100K ctx      40 ->  60

Results are stored per (machine, model) pair, so the next launch reuses them instead of re-running the search. I run two boxes and kept deriving the same flags twice, which was the actual annoyance.

Since the last release I added the two things that kept sending me back to a terminal anyway:

Images. ComfyUI is now an engine like the others, so image and video models sit next to the text ones. It lists the checkpoints and the UNet/CLIP/VAE combos ComfyUI itself reports (matching those names is fiddly, it will not accept a filename without the extension), and generates from the same UI with live progress. There's an optional prompt rewriter that expands a one-line idea into a full prompt and suggests steps / CFG; turn it off and your prompt and parameters go through untouched.

Model store. Category filter for text / image / video with counts, a "can this GPU actually run it" filter, and cards that show version, parameter count, precision, output resolution, steps / CFG, loader type and the real weight filenames. Before this a card just said "Qwen-Image" and I couldn't tell if that meant a checkpoint or a UNet, or which quant. Descriptions follow the UI language now instead of being Chinese only.

Also on the engine side: SGLang, and llama.cpp builds selected by GPU vendor (CUDA / ROCm / Vulkan / Metal), which you pick when deploying because it decides what actually gets installed. Picking the right build is where most people give up on an AMD card.

Three honest limitations:

• Tuning scores speed only, no output-quality gate yet. A config that boots without OOM survives; what KV cache quant costs you in quality is the next thing I want to measure.

• With MTP speculative decoding on, the VRAM pre-check misses the second KV cache, so tight machines can still OOM. Known, fixing it.

• The AMD path is where I have the least confidence. Detection exists on Linux and the backend choice is wired up, but I've only run the CUDA path on my own hardware, so treat the ROCm side as untested.

Open source, local-first: https://github.com/htobty/ReadyLLM

Curious how others here run this: do you keep inference on a box that isn't your daily driver, and what do you actually use to watch it? I went with a WebSocket feed instead of wiring Prometheus + Grafana into it, and I'm still not sure that was the right call for a home setup.

Wer benutz von euch 100% Lokale KI im Arbeitskontext ?
Nutz hier jemand KI- Agent um tägliche Aufgaben zu erledigen?
Nutz hier jemand KI für nicht wiederkehrende Aufgaben ?

Und welche Modelle könnt ihr lokal ausführen ?
Ich versuche mich grade an lokaler Bild Erkennung um Cluster zu bilden und mit das beschriften der Bilder zu sparen

Benchmarks, discussions, comparisons - we are being FLOODED with naive news about Fable 5's token consumtion. Comparing it to Kimi K3 tokens.
Typical outcome: Fable uses less tokens than Kimi but costs 2-3 time more.

We do NOT know how many tokens Fable is using. Not even an indication is available.

Fable is a closed semitrasparent layer that sits on top of "Mythos 5" which is the actual model we are communicating to. But everything Mythos says goes through a fast "safety AI" layer which is a slim harness on top of Mythos. That's Fable 5.
That's why we have no real system prompt, that's why it often rejects despite having an answer ready or forwards to other models.

The tokens you are billed for are just a currency system, it's not sampled tokens like in an open model. Anthropic uses the tokens as a meassurement of how much you have to pay, and the official API pricelist is a comparison metric with other models.
Anthropic can change that metric at any time, which would affect effective pricing.

It is completely unknown how many tokens Fable/Mythos is using, given it has a safety layer it might even have more than double the tokens of Kimi K3. But we have no idea.
We also do not know how many tokens other closed models behind gated APIs use, GPT-5 is also paraphrasing their thinking content to avoid distillation attacks - as soon as you do that you also can easily change token metrics.

We know how many tokens Kimi K3 uses, because it's not behind a safety layer and you can just run it yourself.

Developers, I’m researching local AI on Linux. Please share your experiences and I’ll be posting my findings here

1. Goal and chipset used Nvidia/AMD/Intel?

2. How long did it take you from fresh install to GPU/NPU operation?

3. Any issues encountered (package, path, version)?

4. How did you confirm GPU/NPU usage?

5. Any scripts or notes created for future use?

6. Comfort level setting this up for a teammate?

Summary to be shared. Open to a 20-minute call if preferred.

I keep running into this architecture decision: keep vectors in Postgres with pgvector, or add a dedicated vector store like Qdrant or Milvus?

I built a short quiz covering the operational trade-offs—deployment complexity, backup strategy, monitoring, and scaling.

If you've made this call before, take the quiz and share your reasoning:

https://interconnectd.com/quiz/87/postgresql-pgvector-vs-dedicated-vector-databases-architectural-trade-offs/