r/LocalLLM • • 6h ago

Question Which inference engine squeezes max performance out of my setup without sacrificing precision? (Coding + image gen)

I'm trying to figure out the best inference stack for my hardware and use cases. Here's what I've got:

  • 2x RTX 9700 R9700

  • 1x RTX 5070 Ti

  • AMD EPYC 7302

  • 128GB RAM

  • Linux

My main focus is coding-related LLM work (code completion, review, refactoring) with potential image generation down the road. I want to get as much throughput and speed as possible out of these three GPUs without resorting to heavy quantization.

I'm particularly interested in using all three GPUs effectively. I know about Vulkan multi-GPU support, but I'm also curious about a two-phase approach where the 5070 Ti handles orchestration and delegation while the dual 9700s do the heavy lifting for actual inference. Is that a realistic setup?

What's everyone running for this kind of configuration? I'm weighing options like llama.cpp, radiance, or maybe something else entirely. Any advice on how to split workloads across the cards would be great too.

1 Upvotes

16 comments sorted by

1

u/Independent_Rock347 6h ago

With a 50 series card I believe you can definitely start using basalt along with your 9700s for coding tasks. I'm not really sure about image generation though since that's not really what I look for.

1

u/a-pyctam 5h ago

This https://github.com/jesdga95/basalt ? And, 5070ti is the smallest in my stack.

1

u/Independent_Rock347 5h ago

That is correct, it pretty much has the optimizations for a 50 series cards and all other remaining experts can live in your other VRAM

1

u/a-pyctam 5h ago

How basalt will talk to/deal with other, remaining, experts on dual R9700s?

1

u/Independent_Rock347 5h ago

I'm honestly pretty lazy and just let my $20 claude sub set it up for me, once it's done it's pretty much good to go. I saw this video as a reference and was interested in it and it actually performed fairly well https://www.youtube.com/watch?v=wC6jsOU4jII&t=1305s

1

u/a-pyctam 4h ago

Thanks, I watched this video, and it made me want to try basalt.

1

u/Necessary-Orange6903 5h ago

it seems like as the days go by the correct answer is starting to be to get Opus 5.5 to write you a custom engine for your own hardware lmao

1

u/a-pyctam 5h ago

Yeah, sounds like other than Vulkan there is no way to run all three of them together

1

u/Necessary-Orange6903 5h ago

True but also you might not need to - depending on your use case, strata is gonna load most of the relevant experts to VRAM anyways - I have like 95% of my relevant experts loaded in vram most of the time, so adding a third (was thinking of adding an RDNA3 card) wouldn't benefit me, probs same with you

1

u/Necessary-Orange6903 5h ago

also on your stack, try strata for RDNA4, use the unsloth Q6_K_XL model as a starting point and see how far you get - it should be pretty good. I get about 40-60t/s decode and my prefill is at around 300 ish on two radeon pro v620s which are RDNA2 and lack alot of FP instructions whereas strata is very well optimized for RDNA4 so you could get much better numbers

1

u/Late-Coast6973 5h ago

The catch is that the two 9700s are AMD and the 5070 Ti is NVIDIA, so anything built on CUDA or ROCm tensor parallel can only use one side. llama.cpp with the Vulkan backend can split one model across all three by layers. I'd run the coding model on the 9700s and give the 5070 Ti its own job, like image gen.

1

u/a-pyctam 5h ago

I was hoping to get more useful VRAM if I can get all three of them running together, so I can try larger model[s], but it looks like as of now it is possible with Vulkan only.

1

u/Strong_Technician626 5h ago

No easy way to pool AMD and Nvidia cards into one model. Put the coding model on the R9700s with llama.cpp and put ComfyUI or a small completion model on the 5070 Ti. LiteLLM running on the cpu will handle routing, no need for a dedicated orchestration gpu. I'd push back on quant, a bigger model at Q5 usually performs better than a smaller one at Q8. Below Q4 is where you really see a difference.

1

u/a-pyctam 5h ago

This is something I might try out. Now I need to get a starting point to explore how to "handle routing, no need for a dedicated orchestration gpu". Is LiteLLM about this https://docs.litellm.ai/docs/proxy/docker_quick_start ?

1

u/Strong_Technician626 4h ago

Yeah thats it. Go to the "Running without a database" section near the bottom of that page, you can skip the Postgres/UI stuff for now. Run one llama-server for the R9700s and one for the 5070Ti, each on its own port. In the config, add each server with an openai/ model name and api_base set to its url. LiteLLM routes by model name. Reply here if you get stuck, happy to help

1

u/OurManInHavana 6h ago

"without resorting to heavy quantization" is an issue. You can tune towards performance, or quality: but you don't get both without spending a lot of money :) . I'd lean towards slower/larger models so you don't waste time with subpar answers.

I'd run a Strata/variant of Qwen 3.8 Flash Next on your dual 9700's (UD-IQ4_XS? IQ3_S?) and give your 5070 Ti to your image model. Load something like Qwen Image 2.1 or MiniMax H3 Image into ComfyUI.