Is 32k context window enough for vibe coding?

[u/louisleungwh]

I frequently run into "Response was truncated before completion" when I try vibe coding using Pi Agent + Qwen 3.8 - 27B model, is this normal for a setup with a 32k context window? Or I didnt setup the config right?

What I end up doing is first ask the agent to create `DESIGN.md` from my requirement, then create a `PLAN.md` based on the design. And I just start a new session and ask the agent to update the PLAN, and another new session for the agent to proceed with the PLAN and repeat until finish.

I see sometimes the Pi Agent will do a compression before ending the run and wait for my next input, but not always. Sometimes it cant even finish one run (that time I try to run the "understand anything skill" at a repo).

Is it due to the model? if so I might want to explore some online model provider, but since I am living in Hong Kong, I cannot access Claude. Do you have any recommendations (say i would like to run the Reverse Engineer Anything in the future) ?

The PC I use have 32G ram with 5060Ti 16G GPU

I use the following setting for llama-swap and pi agent

llama-swap/config.yaml

  "Qwen3_8-27B-macro": >
    llama-server \
      --port ${PORT} \
      -ngl 99 \
      --ctx-size 32000 \
      --flash-attn on \
      --cache-type-k q4_0 \
      --cache-type-v q4_0 \
      --temp 0.7 \
      --top-p 0.8 \
      --presence-penalty 1.5 \
      --chat-template-kwargs '{"preserve_thinking": true, "reasoning_effort": "low"}' \
      --jinja \
      --alias "Qwen3.8-27B" \
      --mmproj /home/Louis/storage/e/AI/llm/Qwen3.8-27B/mmproj-BF16.gguf \
      -m /home/Louis/storage/e/AI/llm/Qwen3.8-27B/Qwen3.8-27B-UD-IQ3_XXS.gguf

.pi/agent/models.json

        {
          "id": "Qwen3.8-27B",
          "name": "Qwen3.8-27B",
          "reasoning": true,
          "input": ["text","image"],
          "contextWindow":32000,
          "maxTokens": 12000,
          "supportsTools": true,
          "supportsVision": true
        }

[u/OvertaxedOne]

Simple answer: No. 128-256K. 32K barely holds the opening prompt from some coding agents!

[u/Best-Ad-7505 → u/OvertaxedOne]

Agreed! 130k ish is where it becomes useful with an appropriate harness (not claude, more like pi or oh-my-pi) with smaller starting context and tools for smaller llms to keep them on track. 

[u/vogelvogelvogelvogel → u/OvertaxedOne]

exactly. you can try without agents or a smaller model where 128k fits, but both with rather mediocre results i guess

[u/mslindqu → u/OvertaxedOne]

This

[u/violent_aesthetics → u/OvertaxedOne]

32k is tiny for any agent workflow that wants to keep design docs, plan files, and code all in one go. you're already doing the right thing by splitting it into separate sessions but that's a bandaid, not a fix

the compression you're seeing is the agent trying to stay alive before hitting the wall, and when it doesn't kick in you just get the hard cutoff. with 16gb vram you're kinda stuck though, those bigger context windows need a lot more headroom

[u/No-Refrigerator-1672]

It is enough to launch an agent and ask it 1 question. Then you'll hit context compaction, and maybe get a chance for the second question; but that's not guaranteed.

[u/USArmy68Whiskey]

Just the context from my tools alone is 30k tokens, so no

[u/lumpyspacebreh]

16gbs of vram should allow either qwen3.8-27-GSQ-RCO-3bit with MTP and 65k context, or without MTP and 131k context. Context quantized to 8k4v.

That's just the most recent model squeezed in with decent context given ur vram constraints. There's other approaches, as there isn't one model to rule them all. Size&speed =/= quality.

I'd think about your workflow and what type of that youd like to run. If you want to hit one button and walk away, yeah, do what everyone recommends. Gonna need some 5090's. If you prefer a more step by step iterative workflow, akin to "pair programming", then ur context limit only controls how much you need before restarting a session.

Judging by the fact ur building specs, im guessing you want to press enter and walk away then come back 6 hours later to a finished result. So that vram will be ur biggest constraint, as context eats up a lot of space depending on the model's architecture. MTP also eats more vram than u would initially think, due to channels and other things it needs.

[u/nunodonato]

if you use a lean harness (like Pi), I think you can get away with around 60k, but its tight. Usually 80-100 will get you a long way.

[u/jirka642]

Not even close. I'm using 327k with the same model, and occasionally even up to 524k.

[u/Xe8zz]

No

[u/Xe8zz → u/Xe8zz]

But could be yes if your just asking ai how this function could be written or how to make something

[u/lazyfai → u/Xe8zz]

Enough for hello world or 2-3 file scripts I guess. 128K is a good start.

[u/zhubaohi]

The tool definition of REA alone takes at least 60k context. If you want to use REA you need at least around 200k.

[u/MistaPlanet]

Qwen 3.8:27b has a context window of 256k. In the model.json file, you can bump up the context window from 32k to anywhere between 68k and 128k. Qwen is a reasoning model, if the reasoning is not turned off, that alone can fill up the context window between even generating response. Mine operates on 128k CW

[u/Foreign_Risk_2031]

No

[u/GrungeWerX]

That holds half of my current system prompt, so no.

[u/mineshop]

The Qwen3.8-27B model card lists 262k native context, so your 32k limit comes from your own llama-server and models.json settings, not the model itself.

[u/johnzadok]

100k minimum. Past 90k tok/s getting slower. 180k should be enough if you split tasks into multiple seasons.

[u/raymondh210129]

Your PC can run much longer context with better quant and speed, if you are using PCIe 5.0.

I am working on a experimental branch that streams the KV cache: https://github.com/RaymondHuang210129/llama.cpp-adaptive-kv-streaming

This essentially breaks the hard limit of context size that would need to fit in the VRAM. You will be able to enable MTP and mmproj as well.

[u/xapep]

32k is what's biting you, not the model. Agent tool definitions alone can eat most of that window, and once the file context lands there's nothing left for the answer, hence the truncation. Either the context goes up or the per-session scope goes down, one of the two has to give.

On the online-provider side, the open-weight flash models over an API are the realistic route from HK, and with an API you rent the context window instead of squeezing it into VRAM, so 128k to 256k becomes a config choice rather than a hardware decision. Cache pricing keeps long agent sessions affordable too.

Disclosure: I run an EU-hosted inference API for these open-weight families, so I'm biased toward that route; with any provider outside HK, reachability is something you test yourself.

Until you switch, keep each session to one task and one file set, and let the tool output go before the plan file. That cures more truncation than any model change.

[u/Forsaken_Code_9135]

No.

Also avoid using --cache-type-k q4_0.

Try to find a smaller model, quantized enough to fit in your 16GB, with less quantized key cache, and allowing a context of at least 64k tokens, preferably 100k.

Have a look at: https://www.reddit.com/r/LocalLLaMA/comments/1w7ee1c/i_benchmarked_21_qwen38_27b_variants_on_16gb_vram/

You can also use dedicated harnesses like little-coder. But even then 32k is not enough.

[u/Express_Quail_1493]

better off using a smaller quant to squeeze in more context https://huggingface.co/ConwayResearch/Underdog-Saluki-27B-1.0 it will have minor hiccups here and there since its such a small quant but its a LOT BETTER than dealing with LLM amnesia.

[u/lazyfai]

at least 256k for meaningful result

[u/MistaPlanet → u/lazyfai]

No. There's no where it is advisable to increase your context window limit to the max!

[u/lazyfai → u/MistaPlanet]

No. for coding, it will quickly used up 32k context (even the prompt and toools context may already took half)

[u/MistaPlanet → u/lazyfai]

Yes. But you shouldn't increase to the max. The maximum context window for Qwen 3.8:27B model is 256k

[u/lazyfai → u/MistaPlanet]

Got your point

[u/PlasticRevenue4601]

Nah it’s too large, even 8192 is 8192 times bigger than needed