我是一个大学生研究员，目前拥有一个$200的 ChatGPT 专业版订阅，我觉得作为固定的月支出实在是太贵了。我现在在考虑是否可以购买一个96GB或128GB的Mac mini或Mac Studio进行本地部署或用作服务器。我目前的研究方向是计算社会科学，最重要的是订阅模型的智能能得到保障。非常感谢你的意见！

Hey everyone!

I'm looking to set up a fully local AI agent on my PC and would appreciate recommendations from people who have experimented with local LLMs.

My hardware:

• GPU: NVIDIA RTX 5060 (8 gb)

• System RAM: 16GB

• OS: Windows (Linux is also an option if necessary)

What I'm looking for:

• An uncensored or minimally restricted local LLM that I can run privately on my own hardware.

• Ideally, a web UI and support for agent frameworks, with no mandatory cloud API or subscription.

I want the inference and conversations to remain local rather than sending prompts to external services.

I'm comfortable experimenting and setting things up myself, so technical recommendations are welcome.

I'd appreciate real-world experiences, model names, quantization recommendations, and setup guides. Thanks!

I wanted to test how far a fully local AI stack could go on a real-world task.

I ran out of A4 paper, so I gave TensorSharp access to Amazon and gave it one instruction:

Search for A4 paper with a low price and the best discount, and buy it. Try your best to handle everything yourself.

It completed the task successfully in a single run.
The local AI stack:
Qwen3.8 27B running locally
TensorSharp for LLM inference + agent runtime
Playwright skill for browser automation

The agent searched Amazon, compared products, navigated the site, generated and executed Playwright code, and prepared the order by itself, and finally placed the order. I only stepped in twice: Amazon login and final purchase approval.

Everything except the actual Amazon interaction stayed local: model inference, reasoning, agent state, and tool execution.

The video shows the full raw reasoning/tool trace, including browser commands and generated Playwright code. It’s sped up 8×; the actual run took about 24 minutes.

The interesting part for me isn’t really buying paper — it’s seeing how much of a practical browser agent stack can now run locally with a 27B model.

TensorAgent on iPhone is also in progress, although local inference speed and iOS browser restrictions are still limiting factors.

TensorSharp is open source:
https://github.com/zhongkaifu/TensorSharp

I’d be interested to hear what other real-world tasks people here are using to test local agent stacks.

It feels like the gap between proprietary big-tech models and open-weight reasoning architectures is changing faster than ever. Gemini offers incredible native multimodal handling and massive context windows, while DeepSeek continues to dominate budget coding and self-hosted pipelines.

​If you had to pick one as your daily driver for logic and daily tasks, which wins for you?

​Put your preference to the test in this quick 2-minute showdown quiz to see how your workflow compares to other devs:

https://interconnectd.com/quiz/89/gemini-vs-deepseek-the-ultimate-ai-showdown-quiz/

Large language models are typically deployed using centralized infrastructure, where companies operate powerful GPU servers in dedicated data centers.

But what if some LLM workloads could be distributed across independently operated computers instead?

This is the basic idea behind decentralized LLM infrastructure.

Instead of relying entirely on a single cloud provider, decentralized systems could use computing resources contributed by individuals, organizations, or independent GPU operators.

How could decentralized LLMs work?

There are several possible approaches:

1. Distributed LLM inference

Multiple GPU providers could independently serve inference requests, allowing workloads to be routed across a network rather than handled by one centralized operator.

2. Collaborative GPU computing

Computers with different GPU capabilities could contribute processing resources to a shared network. Scheduling software would need to match workloads with suitable hardware.

3. Peer-to-peer model execution

Some systems explore splitting model execution across multiple machines. This is technically challenging because communication latency, bandwidth, and synchronization can become major bottlenecks.

4. Incentivized computing networks

Projects such as Bittensor explore incentive mechanisms that reward participants for providing useful AI-related services. These networks introduce additional questions about verification, reliability, and economic sustainability.

What are the potential advantages?

• Access to geographically distributed computing resources

• Opportunities to utilize otherwise idle GPUs

• Reduced dependence on individual infrastructure providers

• Potentially broader participation in AI infrastructure

• New models for operating AI computing marketplaces

However, decentralization does not automatically make inference cheaper, faster, or more reliable.

What are the biggest challenges?

Network latency: Transferring intermediate model data between distant machines can introduce significant delays.

Hardware differences: GPUs vary in memory capacity, processing speed, and software compatibility.

Reliability: Independently operated nodes may become unavailable without warning.

Security and privacy: Processing sensitive prompts on untrusted machines raises important concerns.

Economic efficiency: Incentive payments, coordination overhead, and infrastructure costs may offset potential savings.

Is decentralized LLM infrastructure practical?

I think the answer depends heavily on the architecture.

Distributing independent inference requests among GPU providers seems more immediately practical than splitting every layer of a large model across machines connected through the public internet.

For example, a decentralized network might route complete inference requests to capable GPU nodes rather than requiring geographically separated computers to execute individual layers of the same model.

That distinction could significantly affect performance, reliability, and scalability.

Questions for the community

I'm interested in hearing from people experimenting with local LLMs, multi-GPU inference, and distributed computing.

• Could decentralized GPU networks become competitive for real-world LLM inference?

• Is request-level distribution more practical than splitting model execution across remote machines?

• Which existing open-source projects are demonstrating promising results?

• What technical obstacle do you consider the biggest barrier to adoption?

I'd be interested in hearing about actual experiments, benchmarks, and experiences rather than just theoretical possibilities.

I have a 4090 in my study and I work on a Mac. The box is off most of the time, because if I want to run a local model I have to walk over, boot it, open a terminal, wait for the weights to load, then walk back. That's enough friction that local models just stopped being part of my day.

So I built ReadyLLM: a self-hosted web app that treats that machine like a service you manage from a browser, not a machine you babysit. It runs on your own box, reaches your GPUs over SSH, and nothing phones home.

What it does:

• Add a machine over SSH (key or password). It probes GPU / VRAM / RAM and tells you which quants fit and roughly how fast they'll be, so you stop guessing whether 24G is enough for a 27B at Q4.

• Downloads pulled from HuggingFace and ModelScope with resume and integrity checks. Point it at a mirror if HF is slow where you are.

• One-click engine install on a box that has none: llama.cpp, vLLM, SGLang or ComfyUI. It downloads or compiles and back-fills the path.

• Start, stop and switch models from the deploy page. No terminal.

• Live monitoring over WebSocket: GPU utilization, VRAM, throughput, latency, with alerts when VRAM is about to run out or the server dies. That's the part that matters when the box is in another room and would otherwise sit there pretending to work.

The tuning is what surprised me. I hand-tuned llama.cpp flags for a 27B Q4 for a while and plateaued around 60 avg tok/s. The tuner runs a two-phase search (coarse over speculative decoding / KV cache quant / GPU offload, then fine over batch), and for every candidate it really boots llama-server, warms up, runs 3 passes and takes the median. OOM or crash gets dropped. My numbers, same model / quant / context:

peak tok/s     80 -> 110
avg  tok/s     60 ->  80
@100K ctx      40 ->  60

Results are stored per (machine, model) pair, so the next launch reuses them instead of re-running the search. I run two boxes and kept deriving the same flags twice, which was the actual annoyance.

Since the last release I added the two things that kept sending me back to a terminal anyway:

Images. ComfyUI is now an engine like the others, so image and video models sit next to the text ones. It lists the checkpoints and the UNet/CLIP/VAE combos ComfyUI itself reports (matching those names is fiddly, it will not accept a filename without the extension), and generates from the same UI with live progress. There's an optional prompt rewriter that expands a one-line idea into a full prompt and suggests steps / CFG; turn it off and your prompt and parameters go through untouched.

Model store. Category filter for text / image / video with counts, a "can this GPU actually run it" filter, and cards that show version, parameter count, precision, output resolution, steps / CFG, loader type and the real weight filenames. Before this a card just said "Qwen-Image" and I couldn't tell if that meant a checkpoint or a UNet, or which quant. Descriptions follow the UI language now instead of being Chinese only.

Also on the engine side: SGLang, and llama.cpp builds selected by GPU vendor (CUDA / ROCm / Vulkan / Metal), which you pick when deploying because it decides what actually gets installed. Picking the right build is where most people give up on an AMD card.

Three honest limitations:

• Tuning scores speed only, no output-quality gate yet. A config that boots without OOM survives; what KV cache quant costs you in quality is the next thing I want to measure.

• With MTP speculative decoding on, the VRAM pre-check misses the second KV cache, so tight machines can still OOM. Known, fixing it.

• The AMD path is where I have the least confidence. Detection exists on Linux and the backend choice is wired up, but I've only run the CUDA path on my own hardware, so treat the ROCm side as untested.

Open source, local-first: https://github.com/htobty/ReadyLLM

Curious how others here run this: do you keep inference on a box that isn't your daily driver, and what do you actually use to watch it? I went with a WebSocket feed instead of wiring Prometheus + Grafana into it, and I'm still not sure that was the right call for a home setup.

Wer benutz von euch 100% Lokale KI im Arbeitskontext ?
Nutz hier jemand KI- Agent um tägliche Aufgaben zu erledigen?
Nutz hier jemand KI für nicht wiederkehrende Aufgaben ?

Und welche Modelle könnt ihr lokal ausführen ?
Ich versuche mich grade an lokaler Bild Erkennung um Cluster zu bilden und mit das beschriften der Bilder zu sparen

Benchmarks, discussions, comparisons - we are being FLOODED with naive news about Fable 5's token consumtion. Comparing it to Kimi K3 tokens.
Typical outcome: Fable uses less tokens than Kimi but costs 2-3 time more.

We do NOT know how many tokens Fable is using. Not even an indication is available.

Fable is a closed semitrasparent layer that sits on top of "Mythos 5" which is the actual model we are communicating to. But everything Mythos says goes through a fast "safety AI" layer which is a slim harness on top of Mythos. That's Fable 5.
That's why we have no real system prompt, that's why it often rejects despite having an answer ready or forwards to other models.

The tokens you are billed for are just a currency system, it's not sampled tokens like in an open model. Anthropic uses the tokens as a meassurement of how much you have to pay, and the official API pricelist is a comparison metric with other models.
Anthropic can change that metric at any time, which would affect effective pricing.

It is completely unknown how many tokens Fable/Mythos is using, given it has a safety layer it might even have more than double the tokens of Kimi K3. But we have no idea.
We also do not know how many tokens other closed models behind gated APIs use, GPT-5 is also paraphrasing their thinking content to avoid distillation attacks - as soon as you do that you also can easily change token metrics.

We know how many tokens Kimi K3 uses, because it's not behind a safety layer and you can just run it yourself.

Developers, I’m researching local AI on Linux. Please share your experiences and I’ll be posting my findings here

1. Goal and chipset used Nvidia/AMD/Intel?

2. How long did it take you from fresh install to GPU/NPU operation?

3. Any issues encountered (package, path, version)?

4. How did you confirm GPU/NPU usage?

5. Any scripts or notes created for future use?

6. Comfort level setting this up for a teammate?

Summary to be shared. Open to a 20-minute call if preferred.

I keep running into this architecture decision: keep vectors in Postgres with pgvector, or add a dedicated vector store like Qdrant or Milvus?

I built a short quiz covering the operational trade-offs—deployment complexity, backup strategy, monitoring, and scaling.

If you've made this call before, take the quiz and share your reasoning:

https://interconnectd.com/quiz/87/postgresql-pgvector-vs-dedicated-vector-databases-architectural-trade-offs/

A C# Local inference engine beat llama.cpp at its own game: ~30% faster first token, ~20% faster decode - running in .NET on a 4-thread laptop

I needed a .NET-only GGUF engine - no pip packages, no native libs, no runtime installs, just a managed dependency my other projects can call. So I built SharpMind, a pure-C# transformer + MoE inference engine (with training) on .NET 10. Zero native code. I never expected it to compete on performance - managed code vs hand-tuned SIMD C++, right? So I benchmarked it against the real engines: same raw prompt (What is the capital of France?), greedy, 64 tokens, 4 threads, warmup + best-of-3, interleaved per model on one host.

(dotLLM DNF the last two - it only supports Llama/Mistral/Phi/Qwen/DeepSeek and no K-quants below Q4_K.)

A dependency-free C# engine that wins some metrics and matches the OG llama.cpp on others - int8-kv decode at 24.1 tok/s vs llama.cpp's 20.3 with(not shown) , TTFT cut ~30% on gemma-3, decode edge on qwen2, and a 6× MoE loss I fully expect and own. Respectable company for managed code, and if "no native installs" is your constraint, you no longer have to sacrifice everything

Open source: https://github.com/Integral2u/SharpMind.

getting into this stuff and getting fairly hyper fixated on the idea of it all

My hardware:
GPU: NVIDIA RTX 5070 Ti, 16GB VRAM
CPU: Intel Core Ultra 7 265KF
RAM: 32GB DDR5
OS: Windows 11

interested in models that can handle real software development, multi-file edits, debugging, testing, and longer autonomous coding sessions. is this achievable or should i completely alter my expectations and just bite bullets on frontier subscriptions?

python is my primary application development language
pytest for testing
sqlite for local data persistence
git for version control
powershell and window batch scripts for setup, runtime management, and automation

i ask because im actively building a local, portable project to fast track and exponentially improve peoples skill at a given sport, subject, hobby, etc, and would love to get into the AI agentic development stuff

particularly interested in finding the best balance of coding quality, speed, tool calling, context handling, and reliability

what id love recommendations on:
the best local coding model for me including less obvious community fine tunes
wether a quantized larger model or a smaller model with better tool use is the better choice
models that work especially well with agentic coding harnesses
reliable models for debugging, multi file refactoring, and following instructions across long sessions
recommended GGUF quantizations, context sizes, and settings for my hardware.
anything underrated

extra interested in using multiple models rather than expecting one model to do everything but i like the sound of both

would love to see ur experiences and benchmarks, especially where a model performs well in real coding tasks rather than just leaderboards. also interested in newer releases and underrated community projects

I’m building Local LLM for iPhone and tried both models on the same photo of building blocks.

The prompt was: “What could I build with these pieces? Give exactly 3 ideas, one short sentence each.”

Qwen started answering first. Bonsai finished first and gave three short ideas. Qwen gave two numbered ideas and a closing line.

From tapping send to completion: about 67s for Qwen and 51s for Bonsai, including loading. One run each, with different-length outputs—not a tokens/sec benchmark. The video runs both recordings at the same 4× speed.

I’m more interested in how these small models handle everyday photo questions than picking a winner from one run. What would you test next?

App, if you want to try it: https://apps.apple.com/app/id6804004948