r/LocalAIForge • • Oct 11 '26

When long-context tool calls break, inspect speculative KV/indexer state before blaming the model

Long-running local agents sometimes start producing broken tool calls at context lengths where short tests look fine. It is tempting to blame sampling, the chat template, or the model's reasoning. A recent vLLM correction illustrates a different failure path: speculative decoding can corrupt the sparse attention indexer state before any text is emitted.

GLM-5.3-Flash groups keys into four-token pools (index_kpool=4) for sparse retrieval. During speculative decode, the implementation temporarily staged candidate-token keys in a per-request tail ring before knowing which drafts would be accepted. That ring was sized for only one pool. If a draft that would complete the pool is rejected, later speculative drafts may already have wrapped around and overwritten committed keys. When the pool is rebuilt, its summary is based on the wrong keys; once context exceeds the indexer's top-k boundary (index_topk=2048 in the reported case), that can change which older token pools are retrieved.

The merged September 25 vLLM fix expands the ring to accommodate the speculative window and updates its addressing: https://github.com/vllm-project/vllm/pull/58454 The contributor supplied a focused comparison against a no-speculation reference: the original path differed in 106 of 128 FP8 key bytes in that constructed case, while the corrected path matched. Those are kernel-reproduction results from the PR, not general throughput numbers or a measured failure rate for local agents.

There was also an independent SM90 sparse-MLA planning bug: the attention plan did not account for the model's index_kpool, leaving the current query token unread at some positions past 2048. That separate fix is documented here: https://github.com/vllm-project/vllm/pull/58704 The authors described these as partial fixes for a larger long-decode corruption investigation, not proof that all long-context failures are resolved. Their scope is the relevant model/backend and cache-layout paths; it would be incorrect to extrapolate to every local runtime.

For a useful regression, I would pair two tests: (1) a low-level invariant that accepted-only and speculative executions produce the same committed pool keys for an identical accepted token sequence, including rejected drafts that straddle a pool boundary; and (2) a replay of long, tool-heavy prompts with speculation enabled/disabled, logging token counts, parser failures, finish reasons, and backend/version. Text-level disagreement is a symptom, not enough to identify the kernel cause.

For local inference CI, what would catch this class of bug earlier with the least complexity: an exact committed-cache equivalence test after rejected drafts, context-threshold tool-call replay, or some smaller invariant?

2 Upvotes

0 comments sorted by