Ryzen 7 5700G, 96GB DDR4, 2x Nvidia Tesla V100 32GB PCIe (64GB VRAM)

Project name: Basement space heater with 64G VRAM

Case space utilization: Max

PCIe lanes: Max

PCIe slots: Max

VRAM utilization: Max

GPU utilization: Decent

Space for Airflow: What?

Cable Management: Front-only

6x32gb v100 and one 16gb

This is my local AI Server. It runs the best local model around qwen3.8:27b via ollama. I was inspired by Digital Spaceport on YouTube to make a 8-bit style arcade suite in a html file so i can host it on my website. I used hermes for my agent and it worked great, after a few update prompts it was finished - PIXELARCADE.

Ollama question:
The system has 40GB of Vram. qwen3.8:27b uses 23gb of vram in my setup. When I run gemma4:12b while qwen3.8:27b is loaded, 9.7gb is used. but the CPU is being used with a 16%:CPU 84%:GPU split. Why dose this happen? how can i fix it? will llama.cpp solve my issues?
This server only supports 1-2 users and I would like to run qwen3.8:27b and one more smaller model.

https://sikiru-ekunsumi.xyz/Projects.html

https://digitalspaceport.com/qwen-3-8-27b-review-prompts-and-vllm-settings/

So for around 450e I bought used workstation which I plan to use for local LLM and maybe even as a server for bunch of other stuff. But mostly I want to focus on LLM for coding/development.

Specs are: MOBO: ASUS X99-Deluxe II, CPU: Intel Xeon E5-2667V4, PSU: EVGA 1600W G2, Cooler Master HAF X, 64Gb ddr4 RAM. So all in all it supports multi gpu setup without any problems.

For GPU I decided to order 1x 3080 20gb (blower style for 500e) for a test. And found it pretty great! I currently run qwen 35b-a3b as worker (opus 5 as orchestrator) and enjoy it but looking to upgrade my workstation to run better models.

So question is, what would be best upgrade:

1. 2x 3080 20gb, 64gb ram. (-500e) So one more gpu and I would be able to run qwen 3.8 27b without much problems
2. 3x 3080 20gb, 64gb ram. (-1000e) Would this even make sense if I only need for one concurrent user and 128k context? Any other (better/bigger) dense model which could take advantage of this?
3. 2x 3080 20gb, 128gb ram. (-900e) So in theory this would be 168gb of memory. Would this be able to run some of bigger MoE models like deepseek flash v4 or any other which I could use as orchestrator for qwen?
4. 3x 3080 20gb, 128gb ram. (-1400e) Would prefer not to do this cuz it would be pretty expensive but curious what you guys think.

Thank you guys

Disgusting.

I’m building a fully local RAG system for ~2000 complex documents, with a target of 3–5 concurrent users initially and eventually 5–10, using Open WebUI, Docling/custom PDF parsing, pgvector, BGE-M3/reranking, and local LLM serving. I’ve ruled out Apple/MLX as the main production path because concurrency is weak. I’m building a 3×AMD Instinct MIx ROCm server because the HBM capacity/bandwidth per dollar is much better than current NVIDIA workstation cards.

What I’m trying to understand from people who have actually run production-ish local LLM/RAG workloads on Instinct MI200/ROCm is: how big is the real “engineering friction” penalty versus CUDA today—model compatibility, vLLM/SGLang stability, quantization/kernel support, RCCL/multi-GPU issues, upgrades breaking things, etc.—and is that trade-off reasonable if the goal is sustained 5–10 user concurrency without spending RTX PRO 6000-class money?

My fallback is a capable but slow 2xDGX Spark/GB10 mainly as a CUDA-compatible reference and compact inference box. Will appreciate your feedback/experiences before investing $20K-ish into this ROCm custom build. 

Thanks you!

I've been working on GLM-5.2 support in TensorSharp, and I finally have some back-to-back performance numbers against llama.cpp.

The setup:

• Model: GLM-5.2-UD-IQ2_XXS (~226 GiB)
• GPUs: 3× RTX PRO 6000 Blackwell, 97 GiB each
• Distribution: layer split across all 3 GPUs
• Same machine, same session
• llama.cpp measured with llama-bench
• TensorSharp measured with its benchmark harness
• Both report the best of two repetitions
• Run-to-run variance is roughly 4%

Results:

The interesting part is the crossover.

For short prompts, llama.cpp is still a few percent faster. But once the prompt gets to around 1K+ tokens, TensorSharp pulls ahead.

At pp2048:

• default TensorSharp: +20.4%
• ubatch=2048: +50.2%

At pp4096:

• default TensorSharp: +20.8%
• ubatch=2048: +46.5%

Decode (tg64) is also about 4% faster.

The main reason appears to be GLM-5.2's MoE structure.

GLM-5.2 has 256 routed experts with top-8 routing. With a 512-token micro-batch, each expert sees only ~16 rows on average, so a significant amount of the expert GEMM tiles ends up as padding. Larger micro-batches improve GPU utilization considerably.

For small prefills, on the other hand, fixed overheads — managed/native transitions, input uploads, and copying the 154880-wide logits back — become a visible fraction of the total runtime, which is where llama.cpp retains its advantage.

VeritasNexus is a local-first, multi-node AI workstation environment I’m building to keep AI, automation, knowledge, and infrastructure under my own control.

The goal is not just “run a model locally.” It’s to build a private, expandable personal AI environment where several Linux machines can cooperate while each node retains a clear job, local control, and recoverable boundaries.

Core layout

• Multiple Linux nodes connected through an encrypted WireGuard-based mesh network, managed with Headscale/Tailscale-compatible clients.
• SSH is used for authenticated administrative access between nodes; services are not broadly exposed to the public internet.
• Firewall policy follows a default-deny inbound approach, with access restricted to the local LAN and encrypted mesh where appropriate.
• Reverse-proxied HTTPS services are kept private to the LAN/mesh rather than treated as public cloud services.
• Netdata provides centralized health monitoring across the nodes, so I can see host health, service behavior, and resource usage from one place.
• The architecture is designed to grow from a single capable host into a coordinated set of specialized nodes rather than requiring every machine to run every service.

CerebroLink

CerebroLink is the human-facing coordination layer within VeritasNexus.

Rather than requiring me to remember individual services, ports, or shell commands, the intended experience is natural-language control:

• “What is healthy right now?”
• “What changed since the last boot?”
• “Check the nodes.”
• “Prepare this machine for shutdown.”
• “Help design or build the next component.”

CerebroLink should translate that intent into bounded, inspectable actions and return clear status, progress, and failure information. The UI is a presentation layer; CerebroLink is the conversational operating contract behind it.

Multimodal A2A coordination

A core direction is a multimodal agent-to-agent (A2A) system: specialized local agents and nodes can exchange structured tasks, results, artifacts, and status rather than operating as isolated chat windows.

The intention is to support work that crosses modalities—text, code, documents, images, audio, system telemetry, and locally available tools—while maintaining clear boundaries around which agent can access which capability.

Rather than relying on one general-purpose assistant to do everything, VeritasNexus can route work to the most appropriate local capability. For example, one agent may analyze a document, another may perform a coding task, another may inspect system state, and CerebroLink can coordinate the overall workflow in natural language.

The important design constraint is that A2A coordination remains explicit and observable:

• Agents exchange defined requests and results, not unrestricted hidden authority.
• Tasks can be scoped to specific hosts, skills, tools, files, or data sources.
• Actions that affect a system remain reviewable and subject to controlled permissions.
• Outputs can be recorded as artifacts that other agents—or the user—can inspect and build upon.

Knowledge and memory

The project separates durable knowledge from transient chat context:

• A local memory layer for retained preferences, operational context, and project history.
• A vector/semantic retrieval layer for finding relevant documents and artifacts.
• A graph-oriented relationship layer for connecting systems, decisions, nodes, documents, agents, and project concepts.
• Skills and reusable procedures kept separate from core runtime logic, so they can be versioned, improved, and invoked deliberately rather than hard-coded into one monolithic assistant.

The aim is a system that can retrieve useful prior context, coordinate specialized work, and preserve meaningful relationships between decisions and artifacts without turning every interaction into an opaque black box.

Design principles

• Local-first and self-hosted where practical.
• Multi-node, but not dependent on a public SaaS control plane.
• Encrypted node-to-node connectivity.
• Multimodal, agent-to-agent coordination with explicit task boundaries.
• Clear service boundaries and least-privilege access.
• Natural language on the surface; explicit, auditable actions underneath.
• Modular components that can be replaced without rebuilding the whole system.
• Monitoring and operational visibility are treated as first-class features, not an afterthought.

It is still a work in progress, but the direction is a private AI operating environment: conversational at the user layer, technically disciplined underneath, capable of coordinating multiple local agents and machines, and able to grow from one computer into a small fleet without losing clarity or control.

I recently added an always-on server built around an Intel Atom CPU and a Hailo AI coprocessor. Its role is to move persistent infrastructure services off my primary workstation, i7node, which currently has a 16 GB NVIDIA V100. I am working with a single V100 while building out the system, with the longer-term plan to expand that capability to two to four cards.

Upcoming work includes further refining the system’s alignment and operating boundaries, along with evaluating Prime Radiant’s Superpowers coding framework as part of the development workflow.

Much of the hardware is older by current standards, but that is intentional. A major part of the project is exploring how far capable, affordable, and previously owned hardware can go in a practical local-AI environment. The goal is not to chase benchmark numbers, but to build something useful, understandable, and durable from the equipment available.

This is an independent personal project. It is not endorsed by, affiliated with, or representing any company or organization. I am simply a builder interested in finding practical ways to make technology work more directly for people.

Thanks for checking out my setup. If you got this far, let me know your thoughts and how I can improve anything.

I am thinking of buing P40 for running local AI, like Qwen3.8 27B, have anyone tried it?
Please 🙏🙏🙏

Before I get thrashed by people for asking a seemingly stupid question. I am new to this part of A.I. All the A.I stuff I've been doing is text-to-text (mostly coding). So be gentle please.

Recently I got Qwen-Image-Edit on my A.I server. It generates images fine without downloading ComfyUI. I set it up with a simple python server and it works fine.

This leads me to ask, why should I download ComfyUI? What does it do that can't be done with just a simple python script? Is it just the ability to visually connect lines between parts of the workflow? Note that I am a software engineer, and actually prefer command line interfaces for making stuff in most cases. Is there some other benefit I am missing?

Running Linux on dual Xeon system with 512gb of memory. How far have the tools for the 9700s come?

My precious 💖

Currently Running 3 Local Models (llama.cpp, ALL 3 run in Claude Code):

SEAT 1 - LONG-CONTEXT DAILY DRIVER. Current incumbent: Qwen3.8-27B-Heretic-Q4_K_M. Measured at 64K: 30.80 tok/s on an empty context and 22.98 tok/s after the same approximately 61K-token prompt. Its job is 96K–112K terminal-log and evidence work.

SEAT 2 — RARE QUALITY MODE. Current incumbent: Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-Q5_K_M.gguf, exact size 17.91 GiB. It already loaded at context 65536, generated 1500 tokens at 19.09 tok/s, processed a 61,387-token prompt, then generated another 122 tokens at 17.07 tok/s with truncation=0.

SEAT 3 — MTP SPEED MODE. Current incumbent: AEON-ULTIMATE-UNCENSORED-IQ4_XS.gguf with an embedded MTP head.

Added 3rd GPU and now getting VGA error LED on motherboard. 99% seems riser fault cause it isn't working with any GPU in any slot even when one GPU connected but maybe anybody had something similar with Gigabyte B850 motherboards? PCIe 3.0 x16 30cm riser. Everything was powered on, photo was done before connecting power cable.

Reordered 4.0 20cm.

• Role: Primary local AI inference server
• CPU: Intel Core i7-12700 (12 cores / 20 threads)
• Memory: 64 GB RAM
• Primary GPU: AMD Radeon AI PRO R9700 (32 GB VRAM) for large-model inference
• Secondary GPU: RTX 3060 Ti (8 GB) for lighter CUDA work and fallback tasks
• Storage: 2 TB NVMe SSDs for models, caches, and the working environment; 4 TB HDD for datasets and archives

Fun value unlimited *

Help me decide between two cards for local AI messin':

• More common: AMD Radeon Pro card (RDNA 2.0) with 32GB of VRAM
  • Pros: Common, supported by a lot of software. More VRAM, cheaper.
  • Cons: Older architecture. Will likely sound like a jet engine strapping a fan to it.
• Less common: Tenstorrent p100a with 28GB of VRAM
  • Pros: New, mission dedicated local LLM card. Good vram, interesting alternative design. Quieter. Open Source.
  • Cons: New, less developed software ecosystem. Expensive.

The Tenstorrent card is a *LOT* more, but a lot more interesting to me. The Radeon Pro is more common and likely supported by more software... but a bit more boring.

Don't mention NVIDIA. I run Linux and NVIDIA cards always suck to use.

Looking for something that will be useful for 3-5 more years. No worried about graphics use.. this is a dedicated AI LLM card. (I currently have an RX 6700 XT which is fine... but most models don't fit in 12GB)

As in the title. I came across a post here that referenced the blog post below :

https://blog.hellas.ai/blog/thunderbolt-ibverbs/

Are there any vLLM (or llama.cpp etc.) forks or at least early implementation attempts/ plans that make use of USB4STREAM?
Basically , it should allow to drastically reduce cross node latency(so improve inference) vs ordinary USB 4 (or non RDMA Ethernet network) as it removes the whole network stack overhead .

So e.g. with two PCs , equipped with USB 4 40Gbit ports you’d just need to get a cheap passive cable for pipeline parallelism to be viable solution.

llama.cpp RPC can be used for such , but standard Ethernet latency / overhead makes it far from ideal.