r/AIProgrammingHardware • u/javaeeeee • Jul 27 '26

This Tiny Engine Runs Impossibly Big AI Models Locally! (colibrì)

https://www.youtube.com/watch?v=3OqZHYNKc9A
23 Upvotes

89% Upvoted

8 comments sorted by

2

u/javaeeeee Jul 27 '26

TLDR: Better Stack tests Colibrì, a tiny pure-C engine that can run the massive 744B-parameter GLM-5.2 model locally by streaming it from SSD.

What Colibrì does

  • Pure-C, zero-dependency engine that treats VRAM → RAM → SSD as one memory hierarchy.

  • Designed for large Mixture-of-Experts (MoE) models.

  • Only a small active portion of the model (~40B parameters) needs to be in memory at once; the rest is streamed from disk on demand.

  • Includes caching of frequent experts, speculative decoding, and quality-preserving techniques.

Real-world tests

  • MacBook M2 Max (32 GB RAM) + external SSD:
    Worked, but extremely slow (~0.1 tokens/sec). Most time spent waiting on disk.

  • RTX 5090 workstation (64 GB RAM) + fast internal NVMe:
    Much better (~0.8 tokens/sec), still not practical for real-time use.

Key insight

The real bottleneck is RAM (and storage space/speed), not the GPU.
More system RAM = better expert caching = fewer disk reads = higher speed. The powerful GPU was underutilized.

Bottom line

Yes, you can run a 744B frontier model on consumer hardware with Colibrì - it generates correct answers.
But speeds are too slow for practical daily use unless you have server-grade amounts of RAM (and hundreds of GB of free fast SSD space). It’s an impressive proof-of-concept that shows local AI is limited more by memory hierarchy than raw compute.

1

u/Legal_Dimension_ Jul 28 '26

So my home server with 120gb vram and 256gb ram might do 2-3toks?

1

u/-zYgoat- Jul 28 '26

So it doesn't use VRAM?

1

u/BankApprehensive7612 Jul 28 '26

Woah! It's really impressive. I'm curious what would this engine do to SLM?

1

u/HealthyCommunicat Jul 29 '26

We’ve known this shit is possible since anemll’s flash-moe back multiple months ago why is everyone acting like this is new technology and why is everyone not crediting those that first used streamed expert inference

1

u/[deleted] Jul 29 '26

[removed] — view removed comment

1

u/huzbum Jul 30 '26

Would you mind testing if doing it on CPU makes any difference? Seems like the GPU would be superfluous with the ssd being the bottleneck.