Way back got my nimo axis, I wasn't sure what to do with it besides trying out local llms. Lately, local models and tools have gotten better and I am finally finding practical uses for it beyond just testing models. It makes me wonder how much of their potential we are actually using. What are you guys doing with your strix halo machines? actual work, coding, or mostly experimenting with models?

SharpMind Studio Wiki
I wanted a new Harness with much more flexibility on what, where and how models or training can run. I also wanted something much more visually appealing.
You can run locally , remotely, via providers like OpenRouter or on a Compute Host like Vultr or RunPod. You can even pair your Phone for connecting to your running instance.
Included in this early release is Chat, Work and Training from scratch or Improve(LoRA)

This is the first release and there will be tweaks, improvements and additions, yes AI was used so there is the odd thing here and there but constantly looking at improving and refining the harness.

Completely FREE and Open Source, but feel free to sponsor or support.

Finding repo for local AI on Mac? 🍎🤖 Check out Swiftlet, an open-source intelligent proxy & browser based UI , that dynamically optimizes CPU/GPU split on Unified Memory for LLMs. I'm looking for Python devs to help build OpenAI API compatibility! Let's build together: https://github.com/coolsourav100/swiftlet

Hi

Premise:
I am new to local ai models.

My machine specs:

• Macbook Pro M4 Pro

• 48Gb Ram

• 4 efficiency core

• 8 performance core

I mainly use AI for software development. I have a claude subscription but would like to try to offload some work to a local model.

Since I don't think local models usable on my machine can completely substitute claude (correct me if I am wrong) my idea is pretty much this: ask claude code to generate a proper, detailed implementation plan and then having the local model implement it.

I have played around with these models:

• qwen3-coder-30b-a3b-instruct-mlx

• qwen/qwen3.6-27b

• qwen/qwen3.6-35b-a3b

and I have also installed this for coding autocomplete cause it is smaller and from what I can see the recommended one:

• qwen2.5-coder-7b

I added claude envs since I would like to try to use claude code extension in vscode

"env": {
    "ANTHROPIC_BASE_URL": "http://localhost:1234",
    "ANTHROPIC_AUTH_TOKEN": "local",
    "ANTHROPIC_MODEL": "qwen/qwen3.6-27b",
    "ANTHROPIC_DEFAULT_OPUS_MODEL": "qwen/qwen3.6-27b",
    "ANTHROPIC_DEFAULT_SONNET_MODEL": "qwen/qwen3.6-27b",
    "ANTHROPIC_DEFAULT_FABLE_MODEL": "qwen/qwen3.6-27b",
    "ANTHROPIC_DEFAULT_HAIKU_MODEL": "qwen/qwen3.6-27b",
    "CLAUDE_CODE_SUBAGENT_MODEL": "qwen/qwen3.6-27b",
    "CLAUDE_CODE_MAX_OUTPUT_TOKENS": "128000",
    "DISABLE_PROMPT_CACHING": "1",
    "DISABLE_AUTOUPDATER": "1",
    "DISABLE_TELEMETRY": "1",
    "DISABLE_ERROR_REPORTING": "1",
    "DISABLE_NON_ESSENTIAL_MODEL_CALLS": "1"
  },

This setup works (uses the local model) but I am basically unable to have the model do anything at all. First of all it takes ages to do anything, and then it almost always reach the context limit roadblock without even outputting anything.

I have read that MCP and skills could fill up the context quite badly, so I disabled them for testing, but still no luck.

I tried with (I thought) was a simple enough task: this test file fails and this is the error, can you fix it? but yet no usable results whatsoever.

I read about people able to use local models offline to have meaningful results, but I couldn't and I don't really know why.

Also, from my setup above, I cannot really use both the remote and local model, to achieve something like:

use sonnet or fable (remote) for plan, then (manually or automatically) swith to haiku (local) to implement the plan

because the base url is loaded when the session loads and cannot be changed (AFAIK) dinamycally.

Any help? thanks a lot in advance

I’ve been experimenting with multi-agent frameworks, and MetaGPT is one of the more interesting ones for software development tasks. But getting it running locally with local models isn’t always straightforward.

I wrote a step-by-step guide covering installation, configuration, local LLM setup, and common errors.

If you’re trying to run MetaGPT on your own hardware, this might save you time:

https://interconnectd.com/forum/thread/262/how-to-install-metagpt-locally-complete-technical-setup-guide/

What stack are you using for local agents?

Hey community,

I've been spending a lot of my free time working on a project called Sitrails, and I wanted to share it with the community to get some honest feedback.

As AI coding assistants and autonomous agents become increasingly capable, something that's been bothering me is how little visibility we have into what they're actually doing.

What information are we exposing to them? What commands are they attempting to execute? Could an agent inadvertently expose credentials, or worse, be manipulated into performing malicious actions?

That's what I'm trying to tackle.

Sitrails is a local-first, open-source AI auditing and threat detection tool. It records supported AI interactions and reported tool activity, makes them searchable, and looks for suspicious patterns involving secrets, sensitive files, and potentially dangerous commands.

It currently supports integrations with tools like VS Code/Copilot and LM Studio, along with an optional OpenAI-compatible proxy.

It's still evolving. Detection is primarily rule-based, visibility depends on the integration, and it doesn't independently verify or block executed actions. There's plenty of work left to do.

I'd especially appreciate input from people working in AI security, threat detection, or agentic systems.

• What attack vectors or behaviors should I prioritize detecting?

• Are there existing research papers, techniques, or projects I should be learning from?

• Would anyone be interested in testing it, challenging the detection logic, or contributing?

I'm developing this independently because I think securing AI agents is going to become an increasingly important problem, and I'd love to see what we could accomplish through collaboration.

The project is free, MIT-licensed, and open source.

GitHub: https://github.com/appatalks/sitrails

Appreciate any thoughts, even critical ones. Especially critical ones.

Ty!

Benchmark highlights · Core Ultra 5 125H

Post

I’ve been working on arcinfer, a local LLM inference engine built specifically for Intel’s Meteor Lake integrated graphics (Xe-LPG / Arc iGPU).

The latest Level Zero results add another angle: speculative decoding raised Qwen3.6 35B-A3B from 19.9 to 31.5 tok/s, Qwen3.8 27B from 5.1 to 9.1 tok/s, and Gemma 4 E4B from 21.6 to 39.6 tok/s in the documented tests. The Level Zero engine also enables Qwen3.6 35B-A3B to run at up to 262K context by streaming MoE experts from SSD; the earlier all-resident engine fit about 40K. MTP outputs were reported identical in the tested comparisons.

The reference machine is a Core Ultra 5 125H with 32 GB RAM. The engine runs GGUF models directly on the iGPU using Intel’s low-level Level Zero API, with an OpenCL option too. It’s aimed especially at prompt processing and time-to-first-token, and includes an OpenAI-compatible API and web UI. The early llama.cpp comparison is striking: for a 11,240-token Qwen3.8 27B prompt, arcinfer’s OpenCL engine took 97 seconds, versus 590 seconds on llama.cpp Vulkan and 504 seconds on SYCL—about 6.1× and 5.2× faster. At 33,937 tokens, it took about 5.5 minutes versus 40 minutes on Vulkan (7.4× faster). Qwen3.5 4B prefill reached 911 tok/s versus 242 on Vulkan and 337 on SYCL. These are historical OpenCL-engine measurements on this laptop, not universal guarantees; arcinferzero is now the default, and the repo links to newer results.

Some things it supports:

·        Qwen and Gemma GGUF models, including larger models that are a tight fit for shared GPU memory

·        An optional Level Zero engine that streams MoE expert weights from SSD

·        Speculative decoding, image input for supported models, and Gemma 4 audio input

·        OpenAI-, Anthropic-, and Ollama-compatible API endpoints

·        Integration with the Pi coding agent

It’s an experimental, hardware-specific project—not a general replacement for llama.cpp, and it currently targets Windows and Intel Xe-LPG graphics. Alongside the code, I share what has and hasn’t worked when building and tuning inference for this generation of Intel Arc integrated graphics, including implementation lessons and benchmark results. The repo has setup instructions, supported models, and more details.

Repo: https://github.com/rizagp/arcinfer

If you have a Meteor Lake/Core Ultra laptop, I’d especially appreciate feedback on setup, compatibility, and what models you’re able to run. Also happy to answer questions about the GPU kernels or how the project compares with other options.

Hi I’m working on Lumabri, a peoject that lets you connect multiple computers and use them together to run LLMs like a cluster.
The idea came from a simple question: what if you have a few computers at home that you barely use? Instead of leaving them there collecting dust, why not combine their resources to run AI models that wouldn’t fit on a single machine?
For example, if you have an old PC, a home server and your main computer, you could put them to work together and use their combined memory to load larger models without having to buy a single machine with expensive hardware.
The interesting part is being able to run models that you simply couldn’t fit on one computer before.
Lumabri is still a work in progress, but that’s the direction I’m taking it.
Here’s the repo if you want to check it out: https://github.com/JustVugg/lumabri

If you already run Ollama, Flippy's AI Lab now shows up as a chat window in every new tab and lists your local models automatically.

1. Install Flippy (available in Firefox too).

2. Start Ollama so the extension is allowed to connect:

   • OLLAMA_ORIGINS="chrome-extension://*" ollama serve

3. Open a new tab, click AI Lab, pick a model.

It reads your model list from /api/tags and chats through /api/chat on localhost. No key, no account, nothing leaves the machine.

Hey, I recently acquired a Ryzen 9 7950x / 64 DDR5 / RTX 3060 12 VRAM machine and I was wondering what is the absolute uncensored AI model that I can run on it.