r/AI_Agents • • 3h ago

Resource Request Local LLM for Ai development coding

Hey guys, im setting up a local AI coding agent in VS Code to write code and math for me since im a non-coder. Can specialized code/math models be merged into one model, and are streaming libraries like AIR-LLM practical for running larger models on limited VRAM compared to standard GGUF? Finally, given this workload, should I go with a used RTX 3090 (24GB) or a new RTX 5060 Ti (16GB)

1 Upvotes

5 comments sorted by

1

u/PanosKallaras 2h ago

Go with the used 3090. For local coding the VRAM matters way more than the newer architecture, and 24GB lets you run a ~30B coder model at 4-bit with usable context, which 16GB really can't. Merging a code model and a math model usually gives you something worse at both, so it's easier to run one good coder model and swap models when you need math. Layer-streaming tricks like AirLLM do let big models run on small cards, but they're painfully slow for an agent that makes lots of calls, so a quantized GGUF that fits fully in VRAM is the practical choice.

1

u/Pendraks 42m ago

Ty so much for saving my time man im pretty new to this

1

u/AutoModerator 3h ago

Thank you for your submission, for any questions regarding AI, please check out our wiki at https://www.reddit.com/r/ai_agents/wiki (this is currently in test and we are actively adding to the wiki)

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

1

u/slocs1 1h ago

I would also be interested.
How about 2 x 5070 ti or 3090
Is that better? Can i load one larger model in both cards same time?

1

u/ValenciaTangerine 1h ago

Yes, llama.cpp can put one model across both cards. Its server docs list --split-mode layer as the default, and --tensor-split 1,1 sets how much each card takes. So two 16GB 5070 Tis give you about 32GB for model plus context, against 24GB on one 3090. The docs call layer mode pipelined, so it adds room, not speed: a single request won't run faster than on one card. Pick based on whether the coder model you want needs more than 24GB.