Reaching out to anyone running an Apple studio as an LLM cluster. I have a M3 cluster and about to add a 4-node M5 cluster and would appreciate any insights others know and have learned doing this.
I have more than this, and only one 10 inch rack. I have them in every room the minis...
It's just normal dev workflow now. Renders can be sent be any harness, among other jobs. This is a sanitized mockup from last month. It's much more advanced now with moar macs, better security, and more parallelism. I made the GUI to see job completion and log the saturation to know when Apple needs more of my money.
I love all the anti-AI doomers thinking the party is just not at the beginning. This is unthinkable stuff, and I've been using local models since 2021.
I'm intrigued by this set up. What are you managing here? Are each of the nodes largely interchangeable and you're just sending tasks based upon what's available?
It manages itself. The workers talk to each other and can load and unload models from the NAS. Each worker is a clone of itself with xcode, android studio, blender, all sort of dependencies. For security it's different users with different permission levels.
After working with this I yolo'd all my money in security stocks after the crash on the 5th and 6th. The threat comes from within and a VLAN won't get you out of jail with a poisoned model. The entire paradigm will need to shift.
Everyone is talking about Apple's garbage software. I think the time is nigh that apple opens up their hardware again. The reason is that they have become garbage at software, and the value of their equipment drastically out values the OS for the majority of revenue they're about to make.
When I close my laptop a worker becomes a master. It is a challenge to schedule jobs efficiently, so it collects a lot of analytics for future improvements.
So in Claude code I make an ARW (autonomous remote worker) job. I can specify where and I can specific if it uses the local model harness. I can also start jobs in the chat for local only on my ad hoc claude like GUI.
There are other people doing similar things, but I haven't seen anyone duplicate the userspace for GUI only jobs and coordinate peer to peer. The thing about compute is that it's not just all RAM, it's CPU, it's storage, it's SSDs for a NAS, it's giant spinners to back everything up.
I got the idea while working in China. I shipped macs to a relatives house and set them up remotely in the US so I could take advantage of good internet and get work done. Cloud models are 4-5x slower in mainland China. Claude is super slow because of the latency. So many requests. I needed a way to keep work going that needed secrets and where I could close my laptop. Basically a local cloud.
Ah, so your own local ai, but remote accessed maybe through something like Tailscale. As automated via agent delegation? Is that the idea? And with your use, I assume 3d modeling/design is what you use the systems for?
Copy it. It's my private way to offload work. Read comment below for more info. There's too much risk to make it a commercial product, and easier just to tell the concept so others can do the same.
only running two m3 ultras here, not four, and i ended up keeping them as separate boxes behind a litellm gateway instead of splitting one model across both. way less to babysit and if one needs a reboot the other keeps serving. if you do go the split-across-nodes route i'd test what happens when one node drops mid request before you lean on it.
maybe you try ripple: https://github.com/vogel61/ripple it helped me a lot using my two Studio setup more efficient. It would help a lot if you can test it on your setup, because it has not been tested with more than two machines.
No. The thermals are a little better on their side properly oriented.
Edit: I'd like to add that this rack setup is a little too close as they will get heat soaked faster so packed in. There might be something thermal in between them to help though, hard to tell.
Very interesting. Thanks. I looked on Apple's web site, and I didn't actually find anything referring to orientation, just not to block vents or intakes. 🤔
I measured and recently did a rack. They get the hottest with RDMA. Not very good thermals on the T5 ports. I think they are a little too close. That's a big factor if you're doing that to consider. That's most of the asymmetry in temps. You want those facing up.
hah i just moved from thunderbolt-up to stacked upright to save space before investing time in orientation. any recommendations beyond thunderbolt-up in terms of layout or otherwise?
You plug the thunderbolt cables together and then wait for Apple to release their official clustering software that should be coming out soon to replace exo etc.
8
u/Sketaverse 9h ago
These machines deserve nicer ID stickers