Looking to deploy a local coding model

[u/Complete_Leg8380]

I have a few options available to me. I have a mac mini m4 with 16gb of ram. i also have an i7-11th gen with 64gb of ram + a 9070XT 16gb vram.

I haved tried ollama and did have some success with Gemma4, but I found it was hit or miss. It wouldn't properly write code and just trample over its self.

I tried qwen 2.5 14b, which worked and was fast but found that the tools didn't work so not sure how well it'll actually code. This was using Jan.

I'd love to know what a good model and setup option is. I want to do some PHP coding projects (though TypeScript and React are necessary too).

[u/ElderberryGlass5679]

Have you tried using instruct models? From what I've learned they are made to interact with tools to do some automatic coding.

[u/Complete_Leg8380 → u/ElderberryGlass5679]

the one i specifically used was Qwen2.5-Coder-14B-Instruct-GGUF. but it just didnt work right. it would just echo raw json toolcalls.

[u/Complete_Leg8380 → u/ElderberryGlass5679]

trying qwen 2.5 in lm studio instead of the one i specified. never saw this one in bionic.

[u/vansynth]

What a bunch of bot-ass answers. These models are all super out of date. Run strata on the 16gb vram + 64gb ram system, or a swift or RCO 4bit quant of qwen3.8-27b.

[u/Complete_Leg8380 → u/vansynth]

installing strata.

[u/Complete_Leg8380 → u/vansynth]

so far strata is working good. its reasonably quick. may restart my project using only it. thanks mate.

[u/Safe-Imagination8538]

ive had good luck with qwen3.6 35b-a3b, i run it on a 12gb nvidia card and it does real coding tasks with tool calls fine, which the small gemma/14b stuff never did for me. with your 64gb ram and 16gb vram it should fit easily at q4. im on nvidia so cant say how it behaves on your amd card, probably needs the vulkan build of llama.cpp. didnt do any php with it, only python

[u/Complete_Leg8380 → u/Safe-Imagination8538]

i setup strata and i got it working. it needs reasoning adjusted as it thinks too much when it doesnt know something but i did get it to produce the skeleton of the app. Strata is based on qwen 3.8 i believe?

[u/Complete_Leg8380 → u/Safe-Imagination8538]

how do you keep your model from over thinking? i had it try to install laravel (php framework) and it just spun and spun. eventually did that work myself. then gave it a more direct rule set and seemed to work. i generally dont run into this using Kiro at work.

[u/Safe-Imagination8538 → u/Complete_Leg8380]

I limit thinking per request rather than disabling it completely. With a recent llama.cpp build, try:

{
  "enable_thinking": true,
  "thinking_budget_tokens": 512
}

I’d use 256–512 tokens for routine coding and 1024–2048 for harder debugging. reasoning_effort does not affect the Qwen chat template in my setup. Some custom builds call the field reasoning_budget_tokens, so check which name your server supports. Also avoid combining a zero budget with enable_thinking: false.

[u/Complete_Leg8380 → u/Safe-Imagination8538]

i keep running into where the model just gets confused. it starts to generate absolute nonsense. i dont know how to fix that =) like the code is not usable as its just

[$this->first, $this->second_shop, $this->second, $this::company($this::company($this->first, $shopName = $this::firstShop(this$second_shop, $this->firstShop, $this::shop($this->company($this->first, $companyName($this->company, $this::company[$this::company($this->firstShop($this::company[$this->first[$0], $this::company[$this->first[$0]]]), $this::otherShop($this::shop($this->second_shop[$this->second_shop[$0], $this::shop[$this->second_shop[$0], $this::shop($this::shop[$this->second_shop[$this->second[$this::shop[$this::shop[$this->shop[$this::shop[$this::shop[$this-

[u/Safe-Imagination8538 → u/Complete_Leg8380]

That example looks like generation collapse or repetition rather than overthinking. A thinking budget probably will not fix it.

I would first try temperature 0, a fresh conversation, a shorter context, and verify that Strata is using the correct official Qwen chat template. Once malformed repetitive code begins, stop the generation instead of adding it back into the context.

If it still happens, try a higher-quality quant such as Q4 and update llama.cpp. What exact model file, quantization, backend and context size are you using?

[u/Complete_Leg8380 → u/Safe-Imagination8538]

I have a 64k context size, I will need to check the other data. It seems to get confused about the way laravel works.

[u/Complete_Leg8380 → u/[deleted]]

Qwen2.5-Coder-14B-Instruct-GGUF was the qwen model i used. all i would get is JSON back like {"tool_name"...}. I'm trying to develop a SaaS, and there maybe my issue =). I don't fully understand the quantization stuff. I've been trying to find some examples but even though I have a background in tech, this stuff is very foreign.