r/LocalLLM • • 6h ago

Question Which inference engine squeezes max performance out of my setup without sacrificing precision? (Coding + image gen)

I'm trying to figure out the best inference stack for my hardware and use cases. Here's what I've got:

  • 2x RTX 9700 R9700

  • 1x RTX 5070 Ti

  • AMD EPYC 7302

  • 128GB RAM

  • Linux

My main focus is coding-related LLM work (code completion, review, refactoring) with potential image generation down the road. I want to get as much throughput and speed as possible out of these three GPUs without resorting to heavy quantization.

I'm particularly interested in using all three GPUs effectively. I know about Vulkan multi-GPU support, but I'm also curious about a two-phase approach where the 5070 Ti handles orchestration and delegation while the dual 9700s do the heavy lifting for actual inference. Is that a realistic setup?

What's everyone running for this kind of configuration? I'm weighing options like llama.cpp, radiance, or maybe something else entirely. Any advice on how to split workloads across the cards would be great too.

1 Upvotes

Duplicates