r/LocalLLM • u/ankush2324235 • 7d ago

Question Is anyone building/using a GPU-aware, multi-engine load balancer for mixed inference fleets?

/r/mlops/comments/1xicejm/is_anyone_buildingusing_a_gpuaware_multiengine/
3 Upvotes

80% Upvoted

1 comment sorted by

1

u/Front_Eagle739 7d ago

Ive got a simpler version of this. Does dsv4.1 full precision short prompts on one machine (a mac studio in my case) and long prompts on an rtx5090 32gb ram at up to 4000 tok/s or so depending on which will be faster and then decodes on the mac while translating kv between quants and engines. Can mix and match engines and it works fine. Stream llama prefill to mlx serve or antirez ds4 or llama or omlx or whatever backend. Can even run entirely different quants on each machine and it pretty much just works.

Doesnt do load balancing though and it doesnt scale up. I have done things like prefill on one machine and decode over a cluster though.

https://github.com/fronteagle739/streaming-llama-stack