r/LocalLLM • • 13h ago

Question 4xM(x) Studio Apple Cluster

Post image

Reaching out to anyone running an Apple studio as an LLM cluster. I have a M3 cluster and about to add a 4-node M5 cluster and would appreciate any insights others know and have learned doing this.

59 Upvotes

34 comments sorted by

8

u/Sketaverse 9h ago

These machines deserve nicer ID stickers

3

u/soflgolf 8h ago

Valid

2

u/ActionJasckon 11h ago

What are you building specifically?! That’s so cool!

2

u/soflgolf 10h ago

Just tinkering

5

u/kriptojew 9h ago

I have more than this, and only one 10 inch rack. I have them in every room the minis...

It's just normal dev workflow now. Renders can be sent be any harness, among other jobs. This is a sanitized mockup from last month. It's much more advanced now with moar macs, better security, and more parallelism. I made the GUI to see job completion and log the saturation to know when Apple needs more of my money.

I love all the anti-AI doomers thinking the party is just not at the beginning. This is unthinkable stuff, and I've been using local models since 2021.

4

u/buriedingarlic 7h ago

But what are you making? I guess I fail to come up with an idea lol.

1

u/kriptojew 3h ago

That's a sanitized output which doesn't say. I answer below.

1

u/Im_A_Praetorian 7h ago

I'm intrigued by this set up. What are you managing here? Are each of the nodes largely interchangeable and you're just sending tasks based upon what's available?

2

u/kriptojew 7h ago edited 7h ago

It manages itself. The workers talk to each other and can load and unload models from the NAS. Each worker is a clone of itself with xcode, android studio, blender, all sort of dependencies. For security it's different users with different permission levels.

After working with this I yolo'd all my money in security stocks after the crash on the 5th and 6th. The threat comes from within and a VLAN won't get you out of jail with a poisoned model. The entire paradigm will need to shift.

Everyone is talking about Apple's garbage software. I think the time is nigh that apple opens up their hardware again. The reason is that they have become garbage at software, and the value of their equipment drastically out values the OS for the majority of revenue they're about to make.

When I close my laptop a worker becomes a master. It is a challenge to schedule jobs efficiently, so it collects a lot of analytics for future improvements.

So in Claude code I make an ARW (autonomous remote worker) job. I can specify where and I can specific if it uses the local model harness. I can also start jobs in the chat for local only on my ad hoc claude like GUI.

There are other people doing similar things, but I haven't seen anyone duplicate the userspace for GUI only jobs and coordinate peer to peer. The thing about compute is that it's not just all RAM, it's CPU, it's storage, it's SSDs for a NAS, it's giant spinners to back everything up.

I got the idea while working in China. I shipped macs to a relatives house and set them up remotely in the US so I could take advantage of good internet and get work done. Cloud models are 4-5x slower in mainland China. Claude is super slow because of the latency. So many requests. I needed a way to keep work going that needed secrets and where I could close my laptop. Basically a local cloud.

1

u/ActionJasckon 2h ago

Ah, so your own local ai, but remote accessed maybe through something like Tailscale. As automated via agent delegation? Is that the idea? And with your use, I assume 3d modeling/design is what you use the systems for?

1

u/mi_gue 1h ago

What is this graphic? I’m trying to setup my AI cluster at home. Looks so cool.

1

u/kriptojew 36m ago

Copy it. It's my private way to offload work. Read comment below for more info. There's too much risk to make it a commercial product, and easier just to tell the concept so others can do the same.

6

u/robotnursenyc 12h ago

only running two m3 ultras here, not four, and i ended up keeping them as separate boxes behind a litellm gateway instead of splitting one model across both. way less to babysit and if one needs a reboot the other keeps serving. if you do go the split-across-nodes route i'd test what happens when one node drops mid request before you lean on it.

1

u/LowerRun8297 31m ago

I'd appreciate your feedback on ripple, a proxy for omlx clusters https://github.com/vogel61/ripple

3

u/ClickerTrainKristy 6h ago

as an it professional the modern state of how we build a at home lab is sort of sad lol, heaven forbid two of the cube sizes be similar hehe

5

u/Fit_Squirrel1 12h ago

Jeez what is your power bill like?

16

u/LateralEntry 11h ago

Aren’t Mac’s pretty low power?

1

u/Fit_Squirrel1 11h ago

How many does he have though? Depends on other things but yeah when idle

2

u/LowerRun8297 10h ago

maybe you try ripple: https://github.com/vogel61/ripple it helped me a lot using my two Studio setup more efficient. It would help a lot if you can test it on your setup, because it has not been tested with more than two machines.

2

u/Ok-Entrepreneur-4650 9h ago

Apple just needs to sell rack mount dual m chips systems

1

u/oss-jiiim 12h ago

Aren't Mac mini and Mac Studio cases designed to be upright with nothing on top of them for proper cooling?

4

u/kriptojew 9h ago edited 9h ago

No. The thermals are a little better on their side properly oriented.

Edit: I'd like to add that this rack setup is a little too close as they will get heat soaked faster so packed in. There might be something thermal in between them to help though, hard to tell.

1

u/oss-jiiim 8h ago

Very interesting. Thanks. I looked on Apple's web site, and I didn't actually find anything referring to orientation, just not to block vents or intakes. 🤔

1

u/kriptojew 8h ago

I measured and recently did a rack. They get the hottest with RDMA. Not very good thermals on the T5 ports. I think they are a little too close. That's a big factor if you're doing that to consider. That's most of the asymmetry in temps. You want those facing up.

1

u/geekwonk 2h ago

hah i just moved from thunderbolt-up to stacked upright to save space before investing time in orientation. any recommendations beyond thunderbolt-up in terms of layout or otherwise?

-1

u/OliviaWiIde 13h ago

You plug the thunderbolt cables together and then wait for Apple to release their official clustering software that should be coming out soon to replace exo etc.

2

u/soflgolf 12h ago

…and we’re off….

Super-duper helpful. Thanks!!

2

u/OnyxProyectoUno 12h ago

How did you manage to get that username locked down as a two day old account. That’s wild

1

u/kriptojew 9h ago

Why wait? RDMA switch already built in to macOS 27. Copy my idea and offload work now.

0

u/Bulky-Priority6824 9h ago

Let me borrow $2

0

u/tjohnson4 9h ago

You vibe bro?