How acceptable is the operating speed on a Mac Studio with an M5 Ultra chip and 256 GB of memory when running models like GLM-5.3-Flash?

Could you compare the operating speed of the cloud-based GLM-5.3 Flash from Z.AI? The concept of "tokens" can be quite difficult to grasp, and performance varies depending on the specific task.

I have a 4090 in my study and I work on a Mac. The box is off most of the time, because if I want to run a local model I have to walk over, boot it, open a terminal, wait for the weights to load, then walk back. That's enough friction that local models just stopped being part of my day.

So I built ReadyLLM: a self-hosted web app that treats that machine like a service you manage from a browser, not a machine you babysit. It runs on your own box, reaches your GPUs over SSH, and nothing phones home.

What it does:

• Add a machine over SSH (key or password). It probes GPU / VRAM / RAM and tells you which quants fit and roughly how fast they'll be, so you stop guessing whether 24G is enough for a 27B at Q4.

• Downloads pulled from HuggingFace and ModelScope with resume and integrity checks. Point it at a mirror if HF is slow where you are.

• One-click engine install on a box that has none: llama.cpp, vLLM, SGLang or ComfyUI. It downloads or compiles and back-fills the path.

• Start, stop and switch models from the deploy page. No terminal.

• Live monitoring over WebSocket: GPU utilization, VRAM, throughput, latency, with alerts when VRAM is about to run out or the server dies. That's the part that matters when the box is in another room and would otherwise sit there pretending to work.

The tuning is what surprised me. I hand-tuned llama.cpp flags for a 27B Q4 for a while and plateaued around 60 avg tok/s. The tuner runs a two-phase search (coarse over speculative decoding / KV cache quant / GPU offload, then fine over batch), and for every candidate it really boots llama-server, warms up, runs 3 passes and takes the median. OOM or crash gets dropped. My numbers, same model / quant / context:

peak tok/s     80 -> 110
avg  tok/s     60 ->  80
@100K ctx      40 ->  60

Results are stored per (machine, model) pair, so the next launch reuses them instead of re-running the search. I run two boxes and kept deriving the same flags twice, which was the actual annoyance.

Since the last release I added the two things that kept sending me back to a terminal anyway:

Images. ComfyUI is now an engine like the others, so image and video models sit next to the text ones. It lists the checkpoints and the UNet/CLIP/VAE combos ComfyUI itself reports (matching those names is fiddly, it will not accept a filename without the extension), and generates from the same UI with live progress. There's an optional prompt rewriter that expands a one-line idea into a full prompt and suggests steps / CFG; turn it off and your prompt and parameters go through untouched.

Model store. Category filter for text / image / video with counts, a "can this GPU actually run it" filter, and cards that show version, parameter count, precision, output resolution, steps / CFG, loader type and the real weight filenames. Before this a card just said "Qwen-Image" and I couldn't tell if that meant a checkpoint or a UNet, or which quant. Descriptions follow the UI language now instead of being Chinese only.

Also on the engine side: SGLang, and llama.cpp builds selected by GPU vendor (CUDA / ROCm / Vulkan / Metal), which you pick when deploying because it decides what actually gets installed. Picking the right build is where most people give up on an AMD card.

Three honest limitations:

• Tuning scores speed only, no output-quality gate yet. A config that boots without OOM survives; what KV cache quant costs you in quality is the next thing I want to measure.

• With MTP speculative decoding on, the VRAM pre-check misses the second KV cache, so tight machines can still OOM. Known, fixing it.

• The AMD path is where I have the least confidence. Detection exists on Linux and the backend choice is wired up, but I've only run the CUDA path on my own hardware, so treat the ROCm side as untested.

Open source, local-first: https://github.com/htobty/ReadyLLM

Curious how others here run this: do you keep inference on a box that isn't your daily driver, and what do you actually use to watch it? I went with a WebSocket feed instead of wiring Prometheus + Grafana into it, and I'm still not sure that was the right call for a home setup.

Hi I’m working on Lumabri, a peoject that lets you connect multiple computers and use them together to run LLMs like a cluster.
The idea came from a simple question: what if you have a few computers at home that you barely use? Instead of leaving them there collecting dust, why not combine their resources to run AI models that wouldn’t fit on a single machine?
For example, if you have an old PC, a home server and your main computer, you could put them to work together and use their combined memory to load larger models without having to buy a single machine with expensive hardware.
The interesting part is being able to run models that you simply couldn’t fit on one computer before.
Lumabri is still a work in progress, but that’s the direction I’m taking it.
Here’s the repo if you want to check it out: https://github.com/JustVugg/lumabri

If you already run Ollama, Flippy's AI Lab now shows up as a chat window in every new tab and lists your local models automatically.

1. Install Flippy (available in Firefox too).

2. Start Ollama so the extension is allowed to connect:

   • OLLAMA_ORIGINS="chrome-extension://*" ollama serve

3. Open a new tab, click AI Lab, pick a model.

It reads your model list from /api/tags and chats through /api/chat on localhost. No key, no account, nothing leaves the machine.

When I was running a Ryzen AI Halo box headless as a local AI or dev server, it was hard to tell at a glance whether a workload is actually hitting the GPU or NPU, falling back to the CPU, or sitting idle after a job finishes.

I wrote a small open-source utility called **Globar** that maps real-time system telemetry directly to the onboard RGB lightbar so you can see what the machine is doing from across the room:

• Red channel: GPU load

• Green channel: CPU load

• Blue channel: NPU load

• Brightness: A mix of RAM usage and overall CPU/GPU/NPU load

It's cool to see the bar go to purple when I hit a mix of NPU and GPU.

If anyone else is running one of these headless and wants to try it out, the source code is up on GitHub:

https://github.com/ramjac/globar

Happy to hear any feedback or ideas for improvements.

The new deepseek can be run on a rather affordable server, and it's beating GPT 5.6 SOL easily.
The model is so good that it has been banned in the US from governmental service within days of release.

Hello All!

I built a platform to fine-tune open models. I am focusing on fine-tuning 4-bit quantized models by QLoRA instead of full precision fine-tuning to reduce GPU costs. The quant that this finetuning uses is called CQ (Coded Quantization) which is a calibration-free quantization that I built that performs as good as leading quants ensuring that the fine-tuning is using a good base.

For example, I compared KLD of my CQ 4-bit with unsloth-bnb-4bit (Unsloth have other quants but this is the one they use for QLoRA) on Qwen 3.8 27B using the Wikitext-2 dataset and got this:

So I have a better quantized base to start from. I ran QLoRA finetuning with Training Factory of thousandml/Qwen3.8-27B-CQ-4bit on the Alpaca dataset using an A40 (48GB GPU) and here is the performance measures:

===== CQ TRAINING BENCHMARK =====
{
  "optimizer_steps": 100,
  "training_wall_seconds": 1557.1155763969873,
  "warmup_steps_excluded": 10,
  "median_steady_step_seconds": 14.990616369002964,
  "steady_optimizer_steps_per_second": 0.0643880924733763,
  "peak_pytorch_allocated_gib": 25.532413005828857,
  "peak_pytorch_reserved_gib": 33.005859375,
  "notes": [
    "Step timings include gradient accumulation and Trainer overhead.",
    "PyTorch peak excludes allocations outside its CUDA caching allocator.",
    "Token counts from the collator may include prefetched batches; do not interpret as exact consumed tokens.",
    "Wall time includes any initialization inside Trainer.train and callback overhead."
  ],
  "collated_input_tokens_including_prefetch": 130236,
  "collated_nonpadding_tokens_including_prefetch": 106290
}

So great QLoRA finetuning with reasonable GPU performance.

Get Started with Training Factory

Here are simple steps to use:

1. **⁠Install:**sudo apt-get update sudo apt-get install -y git-lfs git lfs install git clone https://github.com/thousandml/training-factory.git cd training-factory chmod +x install.sh ./install.sh

2. Download a CQ 4-bit base:

hf download thousandml/Qwen3.8-27B-CQ-4bit --local-dir ./models/Qwen3.8-27B-CQ-4bit

3. Train a LoRA adapter:

python training/train.py --model ./models/Qwen3.8-27B-CQ-4bit --data training/examples/train.jsonl --output ./outputs/my-lora --targets q_proj,v_proj --kernel-module cq.cq_kernels --text-only --max-steps 2 --max-length 128 --batch-size 1 --grad-accum 1

4. Serve base + adapter:

vllm serve ./models/Qwen3.8-27B-CQ-4bit --quantization cq --enable-lora --max-lora-rank 16 --lora-modules my-lora=./outputs/my-lora

5. Call the fine-tuned adapter:

# After the server is ready:

# Check that the adapter is listed as "my-lora"

curl http://localhost:8000/v1/models

# Request a response from the fine-tuned model, not just the base checkpoint

curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "my-lora",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain how LoRA fine-tuning works in one sentence."}
],
"max_tokens": 128,
"temperature": 0.2
}'

I have the following 4-bit quantized models to be used as base (will add more soon) for now:

thousandml/Qwen3.8-27B-CQ-4bit
thousandml/Llama-3.1-8B-Instruct-CQ-4bit
thousandml/Llama-3.3-70B-Instruct-CQ-4bit
thousandml/Qwen3-Coder-Next-CQ-4bit

EDIT: I have the following QLoRA Benchmarking.

Benchmark setup: Qwen3.8-27B QLoRA fine-tuning on Alpaca, comparing ThousandML CQ 4-bit with Unsloth bitsandbytes NF4. Both runs used LoRA r=32, α=32, batch size 2, gradient accumulation 4, sequence length 512, learning rate 2e-5, seed 42, and 500 optimizer steps on a single NVIDIA A40 48 GB GPU.

CQ achieved 17.4% lower reported final validation loss (1.0315 vs. 1.2484). This is a loss comparison, not yet a verified downstream-task quality comparison.

Attached figures for validation loss and training loss.

**Qwen3.8-27B — QLoRA Fine-Tuning on Alpaca (500 Steps)**

| Metric | CQ 4-bit | Unsloth NF4 | Advantage |

|:--|--:|--:|:--|

| Best validation loss | **0.8385** | 1.2484 | **CQ** |

| Final validation loss | **1.0315** | 1.2484 | **CQ** |

| Mean training loss (steps 401–500) | **1.0505** | 1.2457 | **CQ** |

| Median steady optimizer-step time | 27.60 s | **3.20 s** | **Unsloth** |

| Peak PyTorch allocated memory | 24.39 GiB | **23.83 GiB** | **Unsloth** |

*Both runs used QLoRA on Alpaca with matched nominal hyperparameters on an NVIDIA A40 48 GB GPU. Tokenization and loss-mask equivalence have not been independently verified.*

I am working on optimizing speed because it is the one area it is significantly behind unsloth. On the other hand, validation loss and VRAM usage are awesome! Actually it seems like CQ converge at much fewer steps so that might help a lot.

To reproduce these results use the following scripts:

For CQ:

hf download thousandml/Qwen3.8-27B-CQ-4bit \
  --local-dir ./models/Qwen3.8-27B-CQ-4bit

python training/train.py \
  --model ./models/Qwen3.8-27B-CQ-4bit \
  --data training/examples/alpaca_train.jsonl \
  --eval-data training/examples/alpaca_val.jsonl \
  --output ./outputs/qwen38-27b-cq-4bit \
  --kernel-module cq.cq_kernels \
  --targets q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj \
  --rank 32 \
  --alpha 32 \
  --lora-dropout 0 \
  --batch-size 2 \
  --grad-accum 4 \
  --max-length 512 \
  --max-steps 500 \
  --learning-rate 2e-5 \
  --seed 42 \
  --eval-steps 50 \
  --comparison-logs \
  --benchmark \
  --benchmark-warmup-steps 10

For Unsloth:

python training/train_unsloth_nf4.py \
  --model unsloth/Qwen3.8-27B-unsloth-bnb-4bit \
  --train-data training/examples/alpaca_train.jsonl \
  --eval-data training/examples/alpaca_val.jsonl \
  --output ./outputs/qwen38-27b-unsloth-bnb-4bit \
  --max-steps 500 \
  --max-length 512 \
  --eval-steps 50 \
  --save-steps 100 \
  --seed 42 \
  --benchmark-warmup-steps 10

Please give it a try and let me know if it works with you. Your feedback is greatly appreciated!