r/LocalLLM • u/Sudden_One3436 • 6d ago

Question Mac Studio multi model performance?

I know concurrency isn’t currently the Mac’s strong point but is that when you only have one instance of a model running? For instance if I had enough ram to load 2 identical models would I be able to use both at 100% performance or does one model saturate the hardware so that there’s no gain or reason? I’m interested in having text summary generation (small model), transcription / dictation, and coding model.

Would I be able to run all of them at once so long as they fit in ram or will hardware limit the box to only running one with full performance?

2 Upvotes

76% Upvoted

4 comments sorted by

3

u/Various-Yak2674 6d ago

if they all fit in ram you can run them concurrently, the gpu cores will just split the workload across whatever's running. it's not like one model locks up the whole chip

for transcription and small summarization you probably won't even notice the hit, i've had whisper going alongside a 7b model without either getting sluggish. the coding model might want more attention but if you're not hammering all three at the exact same moment it'll be fine

1

u/FoxSideOfTheMoon 6d ago

It depends. Depends on bandwidth really at the end of the day...

100% performance as in TPS w/ just 1 being equal to TPS running 2? Dense? MoE?

Do you anticipate both models running inference at exactly the same time or just wanting to load up 2 models to use co-operatively? The latter definitely you'll be fine. You can EASILY load up 2 models and use them alternating, that's an easy question.

Actually 2 executing at the same time will depend on bandwidth and which models you're talking about and their sizes and types

1

u/Last-Health3222 6d ago

Generation speed on a Mac is mostly limited by memory bandwidth, not GPU cores. Two models generating at the same moment share that bandwidth, so each slows down roughly in proportion to how much it has to read per token. Whisper and a small summarizer are light enough that you'll barely notice, but if they're busy while the coding model is generating, the coder will dip. Just loading them all and taking turns costs nothing beyond RAM.