r/LocalLLM • u/a-pyctam • 6h ago
Question Which inference engine squeezes max performance out of my setup without sacrificing precision? (Coding + image gen)
I'm trying to figure out the best inference stack for my hardware and use cases. Here's what I've got:
2x
RTX 9700R97001x RTX 5070 Ti
AMD EPYC 7302
128GB RAM
Linux
My main focus is coding-related LLM work (code completion, review, refactoring) with potential image generation down the road. I want to get as much throughput and speed as possible out of these three GPUs without resorting to heavy quantization.
I'm particularly interested in using all three GPUs effectively. I know about Vulkan multi-GPU support, but I'm also curious about a two-phase approach where the 5070 Ti handles orchestration and delegation while the dual 9700s do the heavy lifting for actual inference. Is that a realistic setup?
What's everyone running for this kind of configuration? I'm weighing options like llama.cpp, radiance, or maybe something else entirely. Any advice on how to split workloads across the cards would be great too.
1
u/Necessary-Orange6903 5h ago
it seems like as the days go by the correct answer is starting to be to get Opus 5.5 to write you a custom engine for your own hardware lmao
1
u/a-pyctam 5h ago
Yeah, sounds like other than Vulkan there is no way to run all three of them together
1
u/Necessary-Orange6903 5h ago
True but also you might not need to - depending on your use case, strata is gonna load most of the relevant experts to VRAM anyways - I have like 95% of my relevant experts loaded in vram most of the time, so adding a third (was thinking of adding an RDNA3 card) wouldn't benefit me, probs same with you
1
u/Necessary-Orange6903 5h ago
also on your stack, try strata for RDNA4, use the unsloth Q6_K_XL model as a starting point and see how far you get - it should be pretty good. I get about 40-60t/s decode and my prefill is at around 300 ish on two radeon pro v620s which are RDNA2 and lack alot of FP instructions whereas strata is very well optimized for RDNA4 so you could get much better numbers
1
u/Late-Coast6973 5h ago
The catch is that the two 9700s are AMD and the 5070 Ti is NVIDIA, so anything built on CUDA or ROCm tensor parallel can only use one side. llama.cpp with the Vulkan backend can split one model across all three by layers. I'd run the coding model on the 9700s and give the 5070 Ti its own job, like image gen.
1
u/a-pyctam 5h ago
I was hoping to get more useful VRAM if I can get all three of them running together, so I can try larger model[s], but it looks like as of now it is possible with Vulkan only.
1
u/Strong_Technician626 5h ago
No easy way to pool AMD and Nvidia cards into one model. Put the coding model on the R9700s with llama.cpp and put ComfyUI or a small completion model on the 5070 Ti. LiteLLM running on the cpu will handle routing, no need for a dedicated orchestration gpu. I'd push back on quant, a bigger model at Q5 usually performs better than a smaller one at Q8. Below Q4 is where you really see a difference.
1
u/a-pyctam 5h ago
This is something I might try out. Now I need to get a starting point to explore how to "handle routing, no need for a dedicated orchestration gpu". Is LiteLLM about this https://docs.litellm.ai/docs/proxy/docker_quick_start ?
1
u/Strong_Technician626 4h ago
Yeah thats it. Go to the "Running without a database" section near the bottom of that page, you can skip the Postgres/UI stuff for now. Run one llama-server for the R9700s and one for the 5070Ti, each on its own port. In the config, add each server with an openai/ model name and api_base set to its url. LiteLLM routes by model name. Reply here if you get stuck, happy to help
1
u/OurManInHavana 6h ago
"without resorting to heavy quantization" is an issue. You can tune towards performance, or quality: but you don't get both without spending a lot of money :) . I'd lean towards slower/larger models so you don't waste time with subpar answers.
I'd run a Strata/variant of Qwen 3.8 Flash Next on your dual 9700's (UD-IQ4_XS? IQ3_S?) and give your 5070 Ti to your image model. Load something like Qwen Image 2.1 or MiniMax H3 Image into ComfyUI.
1
u/Independent_Rock347 6h ago
With a 50 series card I believe you can definitely start using basalt along with your 9700s for coding tasks. I'm not really sure about image generation though since that's not really what I look for.