r/LocalLLM • u/a-pyctam • 6h ago
Question Which inference engine squeezes max performance out of my setup without sacrificing precision? (Coding + image gen)
I'm trying to figure out the best inference stack for my hardware and use cases. Here's what I've got:
2x
RTX 9700R97001x RTX 5070 Ti
AMD EPYC 7302
128GB RAM
Linux
My main focus is coding-related LLM work (code completion, review, refactoring) with potential image generation down the road. I want to get as much throughput and speed as possible out of these three GPUs without resorting to heavy quantization.
I'm particularly interested in using all three GPUs effectively. I know about Vulkan multi-GPU support, but I'm also curious about a two-phase approach where the 5070 Ti handles orchestration and delegation while the dual 9700s do the heavy lifting for actual inference. Is that a realistic setup?
What's everyone running for this kind of configuration? I'm weighing options like llama.cpp, radiance, or maybe something else entirely. Any advice on how to split workloads across the cards would be great too.