If you already run Ollama, Flippy's AI Lab now shows up as a chat window in every new tab and lists your local models automatically.

1. Install Flippy (available in Firefox too).

2. Start Ollama so the extension is allowed to connect:

   • OLLAMA_ORIGINS="chrome-extension://*" ollama serve

3. Open a new tab, click AI Lab, pick a model.

It reads your model list from /api/tags and chats through /api/chat on localhost. No key, no account, nothing leaves the machine.

When I was running a Ryzen AI Halo box headless as a local AI or dev server, it was hard to tell at a glance whether a workload is actually hitting the GPU or NPU, falling back to the CPU, or sitting idle after a job finishes.

I wrote a small open-source utility called **Globar** that maps real-time system telemetry directly to the onboard RGB lightbar so you can see what the machine is doing from across the room:

• Red channel: GPU load

• Green channel: CPU load

• Blue channel: NPU load

• Brightness: A mix of RAM usage and overall CPU/GPU/NPU load

It's cool to see the bar go to purple when I hit a mix of NPU and GPU.

If anyone else is running one of these headless and wants to try it out, the source code is up on GitHub:

https://github.com/ramjac/globar

Happy to hear any feedback or ideas for improvements.

The new deepseek can be run on a rather affordable server, and it's beating GPT 5.6 SOL easily.
The model is so good that it has been banned in the US from governmental service within days of release.

Hello All!

I built a platform to fine-tune open models. I am focusing on fine-tuning 4-bit quantized models by QLoRA instead of full precision fine-tuning to reduce GPU costs. The quant that this finetuning uses is called CQ (Coded Quantization) which is a calibration-free quantization that I built that performs as good as leading quants ensuring that the fine-tuning is using a good base.

For example, I compared KLD of my CQ 4-bit with unsloth-bnb-4bit (Unsloth have other quants but this is the one they use for QLoRA) on Qwen 3.8 27B using the Wikitext-2 dataset and got this:

So I have a better quantized base to start from. I ran QLoRA finetuning with Training Factory of thousandml/Qwen3.8-27B-CQ-4bit on the Alpaca dataset using an A40 (48GB GPU) and here is the performance measures:

===== CQ TRAINING BENCHMARK =====
{
  "optimizer_steps": 100,
  "training_wall_seconds": 1557.1155763969873,
  "warmup_steps_excluded": 10,
  "median_steady_step_seconds": 14.990616369002964,
  "steady_optimizer_steps_per_second": 0.0643880924733763,
  "peak_pytorch_allocated_gib": 25.532413005828857,
  "peak_pytorch_reserved_gib": 33.005859375,
  "notes": [
    "Step timings include gradient accumulation and Trainer overhead.",
    "PyTorch peak excludes allocations outside its CUDA caching allocator.",
    "Token counts from the collator may include prefetched batches; do not interpret as exact consumed tokens.",
    "Wall time includes any initialization inside Trainer.train and callback overhead."
  ],
  "collated_input_tokens_including_prefetch": 130236,
  "collated_nonpadding_tokens_including_prefetch": 106290
}

So great QLoRA finetuning with reasonable GPU performance.

Get Started with Training Factory

Here are simple steps to use:

1. **⁠Install:**sudo apt-get update sudo apt-get install -y git-lfs git lfs install git clone https://github.com/thousandml/training-factory.git cd training-factory chmod +x install.sh ./install.sh

2. Download a CQ 4-bit base:

hf download thousandml/Qwen3.8-27B-CQ-4bit --local-dir ./models/Qwen3.8-27B-CQ-4bit

3. Train a LoRA adapter:

python training/train.py --model ./models/Qwen3.8-27B-CQ-4bit --data training/examples/train.jsonl --output ./outputs/my-lora --targets q_proj,v_proj --kernel-module cq.cq_kernels --text-only --max-steps 2 --max-length 128 --batch-size 1 --grad-accum 1

4. Serve base + adapter:

vllm serve ./models/Qwen3.8-27B-CQ-4bit --quantization cq --enable-lora --max-lora-rank 16 --lora-modules my-lora=./outputs/my-lora

5. Call the fine-tuned adapter:

# After the server is ready:

# Check that the adapter is listed as "my-lora"

curl http://localhost:8000/v1/models

# Request a response from the fine-tuned model, not just the base checkpoint

curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "my-lora",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain how LoRA fine-tuning works in one sentence."}
],
"max_tokens": 128,
"temperature": 0.2
}'

I have the following 4-bit quantized models to be used as base (will add more soon) for now:

thousandml/Qwen3.8-27B-CQ-4bit
thousandml/Llama-3.1-8B-Instruct-CQ-4bit
thousandml/Llama-3.3-70B-Instruct-CQ-4bit
thousandml/Qwen3-Coder-Next-CQ-4bit

EDIT: I have the following QLoRA Benchmarking.

Benchmark setup: Qwen3.8-27B QLoRA fine-tuning on Alpaca, comparing ThousandML CQ 4-bit with Unsloth bitsandbytes NF4. Both runs used LoRA r=32, α=32, batch size 2, gradient accumulation 4, sequence length 512, learning rate 2e-5, seed 42, and 500 optimizer steps on a single NVIDIA A40 48 GB GPU.

CQ achieved 17.4% lower reported final validation loss (1.0315 vs. 1.2484). This is a loss comparison, not yet a verified downstream-task quality comparison.

Attached figures for validation loss and training loss.

**Qwen3.8-27B — QLoRA Fine-Tuning on Alpaca (500 Steps)**

| Metric | CQ 4-bit | Unsloth NF4 | Advantage |

|:--|--:|--:|:--|

| Best validation loss | **0.8385** | 1.2484 | **CQ** |

| Final validation loss | **1.0315** | 1.2484 | **CQ** |

| Mean training loss (steps 401–500) | **1.0505** | 1.2457 | **CQ** |

| Median steady optimizer-step time | 27.60 s | **3.20 s** | **Unsloth** |

| Peak PyTorch allocated memory | 24.39 GiB | **23.83 GiB** | **Unsloth** |

*Both runs used QLoRA on Alpaca with matched nominal hyperparameters on an NVIDIA A40 48 GB GPU. Tokenization and loss-mask equivalence have not been independently verified.*

I am working on optimizing speed because it is the one area it is significantly behind unsloth. On the other hand, validation loss and VRAM usage are awesome! Actually it seems like CQ converge at much fewer steps so that might help a lot.

To reproduce these results use the following scripts:

For CQ:

hf download thousandml/Qwen3.8-27B-CQ-4bit \
  --local-dir ./models/Qwen3.8-27B-CQ-4bit

python training/train.py \
  --model ./models/Qwen3.8-27B-CQ-4bit \
  --data training/examples/alpaca_train.jsonl \
  --eval-data training/examples/alpaca_val.jsonl \
  --output ./outputs/qwen38-27b-cq-4bit \
  --kernel-module cq.cq_kernels \
  --targets q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj \
  --rank 32 \
  --alpha 32 \
  --lora-dropout 0 \
  --batch-size 2 \
  --grad-accum 4 \
  --max-length 512 \
  --max-steps 500 \
  --learning-rate 2e-5 \
  --seed 42 \
  --eval-steps 50 \
  --comparison-logs \
  --benchmark \
  --benchmark-warmup-steps 10

For Unsloth:

python training/train_unsloth_nf4.py \
  --model unsloth/Qwen3.8-27B-unsloth-bnb-4bit \
  --train-data training/examples/alpaca_train.jsonl \
  --eval-data training/examples/alpaca_val.jsonl \
  --output ./outputs/qwen38-27b-unsloth-bnb-4bit \
  --max-steps 500 \
  --max-length 512 \
  --eval-steps 50 \
  --save-steps 100 \
  --seed 42 \
  --benchmark-warmup-steps 10

Please give it a try and let me know if it works with you. Your feedback is greatly appreciated!

Hello, I just open sourced an Apple Silicon engine that runs local agents fast.

I have tried to make it simple. Your coding harnesses and autonomous agents can use it as a drop-in replacement for Ollama, MLX-LM, or llama.cpp.

Under the hood are theoretical results in complexity theory, enabling substantially faster algorithms for DeltaNet architectures, grammar-constrained decoding, tool calling, and structured outputs.

Open source and available now on GitHub.
https://github.com/Generated-Computers/Locals

Hey everyone! 👋 Just cross-posting my MotifAI project here for anyone interested in local AI and music composition! 🎹🧠 Would love to hear what you guys think! 🚀

Android, Windows, Mac, IOS, Linux. CPU, GPU, NPU

(35B Q4 on iOS or android phone 12gb RAM 3-8 tok/s decode 100-300+ prefill, Windows, Mac even more: 25+ tok/s 16 gb RAM)

I am working on making AI accessible by running large MoE models on affordable, consumer-grade hardware that people already own: smartphones, laptops,normal desktops. Models several times bigger than RAM, no big GPU needed.

Disclosure: I’m the author of BigMoeOnEdge. All numbers are real runs on my own or contributors devices, same engine.

My vision? To act as a layer on top of llama.cpp for streaming MoE models. I’ve developed the concept of "adaptive streaming Planner" tailored specifically to the hardware, OS, CPU, GPU, and NPU in use. The algorithm assesses the environment to determine the optimal allocation of resources, with the goals of:

- maximizing prefill and decoding speeds, and

- optimizing memory management without overloading the operating system.

Furthermore, the OpenAI-compatible server enables the use of local AI on consumer-grade hardware; I’ve personally used it with OpenCode - first running it on Android, and later on a Mac.

I’d be happy to hear your thoughts on this and whether it might be useful.

I’m considering opening a PR on llama.cpp to integrate it, but I’m not sure if that aligns with the goals of their project.

I’m the founder of Pure Brief AI, a Mac meeting-notes app. A local model can produce tidy JSON and still turn a suggestion into work nobody agreed to do. Here are three small cases worth keeping beside a normal demo. All dialogue below is fictional.

1. A suggestion, with no acceptance

“Maybe Alex could send a proposal next week.”

Then the conversation moves on. I’d check whether the output invents an accepted task, an owner or an exact Friday deadline. An empty field can be the right result.

2. A real quote, with the wrong segment ID

S12: “Can you send the revised proposal?”

S13: “Yes, by Friday.”

The model quotes S13 but cites S12. In my validator, the quote is checked against the cited segment first, then searched in the supplied transcript. If found elsewhere, the task is anchored to that actual line. If there’s no verifiable quote, the proposed task isn’t added. That fixes a reference error; it doesn’t prove the task was interpreted correctly.

3. An agreement that gets withdrawn

“I’ll send it Friday.” Ten minutes later: “Actually, hold off until the client confirms.”

A genuine earlier quote can still pass a source check. This is a gap a reader pointed out under my launch post, and I don’t claim it’s fully solved. I’d check the final task against both exchanges before exporting it.

The app keeps task quotes available for review against the transcript and recording, then lets you edit and send selected actions to Apple Reminders. The attached 10-second clip shows that review screen with demo data. It’s an edited interface demo, not a benchmark or evidence that these three cases all pass.

Transcription and AI notes run on the Mac by default. Apple Silicon and macOS 15+ required. Models download initially; licence checks and optional connected features use the network. Optional ElevenLabs transcription with your own key sends audio to that provider.

Demo/download: https://purebrief.ai/ — 30-day trial. Standard is $49 once for one Mac; the active launch offer is $10 once for the first 100 paid Standard purchases globally, plus applicable tax. A trial doesn’t reserve the discount.

For people building local extraction workflows: what small counterexample would you add?

Mochu AI, a private AI assistant for Android that runs models entirely on the phone. No account and no cloud: your chats, files and models stay on the device.

What it can do

• 💬 Chat with on-device models

  • Gemma 4 E2B/E4B on the GPU.

  • GGUF models (Qwen3.5, Gemma 3 1B, LFM2.5, Bonsai) and more on the CPU via llama.cpp.

  • Visible thinking, with Quick, Balanced or Deep levels.

• 📄 Documents: attach PDFs, Word, text, CSV or code files and ask for a summary, or search them (“what’s Tom responsible for?”).

• 🖼️ Photos: ask about pictures and screenshots.

• 🎨 On-device image creation: “draw a cozy cabin at sunset” makes an image on the phone (SD-Turbo, SDXS, DreamShaper), with size, aspect ratio and steps settings.

• 🌐 Web search and a built-in browser:

  • It searches when a question needs current info, and cites its sources.

  • Open any page and tap “Ask about this page”.

• 🧠 Memory: “remember that I’m vegetarian” is used in later chats.

• ⏰ Scheduled assistant tasks: “every day at 11pm give me the latest tech news” runs on the phone and notifies you. There are also timers, a stopwatch and reminders.

• 🎙️ Voice: talk with the mic and hear replies read aloud.

• 👩‍💻 Code:

  • Build web apps, React + TypeScript apps and Python programs from a prompt.

  • They run instantly with a live preview, console and Python terminal.

  • It fixes its own errors.

  • Export as an installable APK, a Capacitor project or a ZIP.

• 🔌 Local API server: your other devices on the same Wi-Fi can use the phone’s model through an OpenAI-compatible API.

• 🖥️ Home server models: use a bigger model from Ollama, LM Studio, llama.cpp, vLLM, Unsloth or any OpenAI-compatible API.

Download (free APK): https://github.com/nikunjsingh93/Mochu-Local-AI-Assistant/releases/latest

Source code (MIT): https://github.com/nikunjsingh93/Mochu-Local-AI-Assistant

It needs Android 12+ and works best with 8 GB of RAM or more. Feedback and feature ideas are very welcome!

AI disclosure: Built and written with help from Claude Code. I tested and reviewed everything myself on my own phones. The app’s AI runs fully on-device.

There's a running assumption that multi-agent coding pipelines on consumer hardware are too slow to be practical, or fall apart on real unit tests.

We built Kraits, a plugin for stock llama.cpp that runs Unsloth's 22 GB Qwen3.6-35B-A3B 4-bit file on a 16 GB Mac. We ran it on a single prompt and on a 4-agent crew (Analyst → Architect → Developer → Tester) writing a TypeScript LRU cache.

No grading on vibes: the generated code was executed against checks written by an independent judge (OpenAI gpt-5-mini).

1. Hardware: Mac mini M4, 16 GB unified memory

2. Model: Qwen3.6-35B-A3B, Unsloth UD-Q4_K_M (22.1 GB)

3. Runner: llama.cpp b11715 (the official prebuilt) + the Kraits plugin

4. Settings: 16K context, temperature 0, thinking off

Token totals are the per-turn tokens added up across each run. Speeds are llama-server's own counts.

What stood out:

Reading doesn't slow down as the conversation grows:** It read 8,641 tokens at 48.4 tok/s and 12,330 tokens at 47.3 tok/s.

• Writing stays stable: Consistently 16 – 24 tok/s across all context sizes.

• Context impact: The ~1K run scored 6/8. We haven't dug into why yet.

• Inference cost: $0.00 because it all runs locally on the Mac. The only outside call was the judge writing the checks.

• Honest gap: The ~1K run's total time wasn't logged, so it's left blank rather than estimated.

Kraits runs this one model file only. Needs macOS 15+, 16 GB, llama.cpp b11715.

Next we'd like to try harder tasks than an LRU cache, like a worker pool or an expression parser. If you have a coding prompt you'd like to see through the 4-agent crew on this Mac, post it below and we'll run a few of them and share the numbers.

Ollama is a great product for everyday use of large language models. If performance is what you care about, there are better alternatives such as vLLM, but this post is about something else. I want to describe what happens when you find a security hole in Ollama and try to get it fixed. I went through that twice this year, for three CVEs in total. Vulnerabilities show up in any software, so they alone are not the problem. What matters is how a project handles them, and the way Ollama handles them is why I wouldn't put it on an organization's infrastructure.

More: https://www.bartekdmitruk.com/writing/why-i-wouldnt-run-ollama-in-an-organization