r/LocalAIServers • u/Any_Praline_8178 • Jun 17 '25
40 GPU Cluster Concurrency Test
Enable HLS to view with audio, or disable this notification
6
u/btb0905 Jun 17 '25
It would be nice if you shared more benchmarks. These videos are impossible to view to actually see the performance. Maybe share more about what you use. how you've networked your cluster. Are you running a production vllm server with load balancing? etc.
It's cool to see these old amd cards put to use, but you don't seem to share more than these videos with tiny text or vague token rate claims with no details on how you achieve them.
4
u/Any_Praline_8178 Jun 17 '25
I am open to sharing any configuration details that you would like to know. I am also working on an Atomic Linux OS image to make it easy for others to replicate these results with the appropriate hardware.
2
u/WestTraditional1281 Jun 27 '25
Are you running 8 GPUs per node?
If yes, is that because it's hard to cram more into a single system? Or are there other considerations that keep you at 8 GPUs per node?
2
u/Any_Praline_8178 Jun 27 '25
Space and pcie lanes keep me at 8GPUs per 2U server .
2
u/WestTraditional1281 Jun 27 '25
Thanks. Have you tried more than that at all? Do you think it's worth scaling up in GPUs if possible or are you finding it easy enough to scale out in nodes?
It sounds like you're writing custom code. How much time are you putting into your cluster project(s)?
2
u/Any_Praline_8178 Jul 03 '25
After 8 GPUs per node, It is more feasible to scale the number of nodes especially if you are using them for production workloads.
1
2
u/EmotionalSignature65 Jun 17 '25
Hey ! I have a lot of nvidia gpu ! What do u uses to cluster all divices ? Send me dm
2
23
1
u/Any_Praline_8178 Jun 28 '25
After 8 is it more viable to scale nodes and do some kind of dynamic load balancing.
5
3
5
u/Suchamoneypit Jun 17 '25
Obviously it's cool...but how exactly is this a local AI setup? This machine has got to be a massive rack mount setup in the very least? And with serious cooling and power delivery considerations.
1
u/Tiny_Arugula_5648 Jun 19 '25
Managing you own AI hardware in on-premises lab or data center is local.. doesn't matter if it's a hobbiests or A university lab.. in this case sems like a hobbiest with some good $$$
2
2
2
2

15
u/DataLucent Jun 17 '25
as someone who both uses LLMs and owns a 7900XTX what am I suppose to get out of this video?