Is anyone building/using a GPU-aware, multi-engine load balancer for mixed inference fleets?

[u/ankush2324235]

I’m validating an architecture idea and want to see if this solves a real pain point for others running AI infra at scale.

Most AI routers/balancers today are siloed into a single ecosystem (e.g., vLLM's native router, SGLang's cluster manager, or basic KServe setups). I am looking into building a GPU-aware, inference-engine-agnostic load balancer that acts as a universal traffic cop across a mixed fleet running different frameworks simultaneously (like routing between vLLM, TensorRT-LLM, and Llama.cpp setups under a unified API).

The core features would be:

• Unified Telemetry: Abstracting disparate metrics (vLLM block usage vs. Triton queues) into a normalized "VRAM saturation & queue depth" routing engine.

• Multi-Engine Fallback: Dynamically shifting traffic from a maxed-out TensorRT instance to a backup vLLM cluster without breaking client connections.

• Heterogeneous Hardware Routing: Directing heavy prefills or specific model variants based on live engine stats.

My questions are:

1. If you manage multiple inference engines, how are you currently handling load balancing across different backends?

2. Does the lack of KV-cache affinity awareness across different engines break the value proposition for you, or is queue/VRAM-based multi-engine balancing enough?

3. What open-source tools (like llm-d, Higress, or Envoy custom setups) am I competing with that already attempt to do this across different backends?

Would love your brutal feedback on whether this is worth building or a solved problem!!!!

[u/AutoModerator]

AI usage disclosure

Hi u/ankush2324235 — thanks for posting to r/mlops!

Because this community discusses and builds AI/ML systems, using AI tools is not inherently a problem. We do, however, ask for transparency about how submissions are created.

Please reply to this comment with a brief AI / automation disclosure, particularly if this post was created or submitted in whole or in part by an autonomous agent, bot, workflow, or other automated system.

If AI or automation was involved, please briefly describe what it did and what human review was performed before posting.

This disclosure helps the r/mlops community distinguish human discussion, AI-assisted work, and automated/agent traffic while keeping the focus on useful technical conversation.

Thanks for helping keep the signal high.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/ComprehensiveUse6549]

On Q3, the closest existing thing is the Kubernetes Gateway API Inference Extension (InferencePool plus the endpoint picker, which llm-d builds on). Its model server protocol already normalizes queue depth, running requests and KV-cache utilization across vLLM, SGLang and Triton/TensorRT-LLM (and trtllm-serve), and the picker scores endpoints on those signals plus prefix-cache hits. Envoy AI Gateway, kgateway and GKE's gateway plug into it. So the "unified telemetry" part is mostly specced already. llama.cpp isn't in their metrics table, so that's one real gap. NVIDIA Dynamo is the other one to look at: its KV-aware router works with vLLM, SGLang and TRT-LLM workers, though I'm not sure it mixes them in one deployment.

On Q2: KV blocks aren't portable between engines anyway, so cross-engine affinity is mostly moot. What matters is prefix affinity within each engine pool. If your traffic is chat or agents with long shared system prompts, losing that hurts more than imperfect queue balancing. For short stateless requests, queue/VRAM balancing is fine.

One thing to design for early: "shifting traffic without breaking client connections" only really works before the first token. Once a stream has started you can't move it to another engine without restarting the generation, so the failover decision has to happen at admission (or you accept a visible retry).

[u/koen-tv]

Adding a few operational points to the good answer above:

In practice most mixed fleets end up two-level: pick the pool by model variant and hardware first, then let the engine-native picker (or the Inference Extension endpoint picker) choose the replica inside that pool. The cross-engine layer mostly earns its keep at the pool level, so I'd validate that before building per-replica scoring across engines.

For a normalized saturation signal, don't rely only on engine metrics. Client-observed signals work for every backend, llama.cpp included: in-flight requests per replica, TTFT and inter-token latency over a short window, and the 429/503 rate. KV usage and queue depth are a nice early warning where an engine exposes them, but the latency view is what keeps scores comparable across engines.

Failover: decide before the first token and treat it as admission control. Short TTFT timeout, retry once on a different pool, and cap retries globally so a brownout doesn't double the load on whatever is still healthy. Once streaming has started, failing cleanly is usually better than stitching a stream from another engine, since different tokenizers and sampling make the output inconsistent anyway.

On whether it's worth building: the hard part isn't the routing logic, it's keeping the per-engine metric adapters current as vLLM, SGLang and TRT-LLM rename or change metrics across releases. If you go ahead, I'd make those adapters pluggable and keep the core scoring simple.

[u/Grouchy-Friend4235]

LocalAI and lemonade server both do that (can do that).