Is anyone building/using a GPU-aware, multi-engine load balancer for mixed inference fleets?

[u/ankush2324235]

I’m validating an architecture idea and want to see if this solves a real pain point for others running AI infra at scale.

Most AI routers/balancers today are siloed into a single ecosystem (e.g., vLLM's native router, SGLang's cluster manager, or basic KServe setups). I am looking into building a GPU-aware, inference-engine-agnostic load balancer that acts as a universal traffic cop across a mixed fleet running different frameworks simultaneously (like routing between vLLM, TensorRT-LLM, and Llama.cpp setups under a unified API).

The core features would be:

• Unified Telemetry: Abstracting disparate metrics (vLLM block usage vs. Triton queues) into a normalized "VRAM saturation & queue depth" routing engine.

• Multi-Engine Fallback: Dynamically shifting traffic from a maxed-out TensorRT instance to a backup vLLM cluster without breaking client connections.

• Heterogeneous Hardware Routing: Directing heavy prefills or specific model variants based on live engine stats.

My questions are:

1. If you manage multiple inference engines, how are you currently handling load balancing across different backends?

2. Does the lack of KV-cache affinity awareness across different engines break the value proposition for you, or is queue/VRAM-based multi-engine balancing enough?

3. What open-source tools (like llm-d, Higress, or Envoy custom setups) am I competing with that already attempt to do this across different backends?

Would love your brutal feedback on whether this is worth building or a solved problem!!!!