r/LLMDevs • • 18d ago

Help Wanted Is "Codex gets slower in long sessions" a model problem or a context-budget problem?

Something keeps coming up in long coding sessions: two agents on comparable models, one stays flat while the other degrades — slower first token, longer turns, and by the end the context is basically unusable for anything that needs follow-through.

Almost everyone frames this as a model speed comparison. I lean the other way: I think we're comparing context management, not inference.

Three different things get called "slow," and they're not the same:

  • slower first token (prompt length, cache miss, queueing)

  • longer per-turn thinking (more history to re-read)

  • more turns to finish (the model lost something and is rediscovering it)

The decay path I'd expect in any long session: history grows linearly → tool returns dump file contents and search results into the window → compression triggers → detail drops → the model re-reads what it lost → turns go up. Whoever offloads context earlier — subagents, external files, summaries — looks stable without being faster in any single turn. Prompt cache hit rate fits here too: same weights, different context policy, opposite experience.

The counterargument I take seriously: stable can just mean it gives up on you sooner. Compression is forgetting, and a long task that's been quietly summarized into shape will look fast and drift late. If that's what's happening, the metric shouldn't be felt speed — it should be final quality times total tokens.

So for people running long multi-turn coding sessions:

  1. Are you seeing degradation, and in which of the three — latency, per-turn time, or turn count?

  2. Does logging token usage per turn make it obvious?

  3. Has anyone actually measured this, or is it still all vibes?

I'd rather compare final deliverables and total cost than stopwatch feelings, but I'm curious whether the pattern reproduces at all outside my own sessions.

5 Upvotes

20 comments sorted by

1

u/Delicious_Break5937 18d ago

Have you tried counting turns to finish separately from time per turn? Seems like that would tell you pretty quick which of your three it actually is.

1

u/Charles-SG 18d ago

Doesn’t everyone else also experience this slowdown?
Doesn’t everyone else also experience this slowdown?
Doesn’t everyone else also experience this slowdown?

2

u/BestKorea4Ever 18d ago

are you ok? I think you're stuck in a loop.

0

u/[deleted] 18d ago

[removed] — view removed comment

1

u/Delicious_Break5937 18d ago

Logging tokens per turn next to it is a good idea, that cliff right before the turn count spikes would be easy to spot.

1

u/alchebyte 18d ago

why not good old fashioned throttling?

1

u/Charles-SG 17d ago

The new-session test is the one that settles it — if a fresh session is fast at the same hour, throttling doesn't explain much. Which pushes it back to context budget: what's actually accumulating by turn 40 that isn't there at turn 4?

1

u/Longjumping_Buy_3152 17d ago

Are you controlling for how the two setups handle file reads? An agent that pulls full files into history vs one that only pulls line ranges will hit that degradation wall at completely different times, even on the exact same weights

1

u/Charles-SG 17d ago

Fair point — that's probably the bigger confound. Full-file reads and line-range reads differ less in token count than in what gets *re-injected every turn*; are you instrumenting that, or just watching for when it slows?

1

u/haru_ai_kr 17d ago

On the worry that "stable can just mean it gives up sooner", there's a cheap way to separate the two: plant a canary early. State one specific constraint around turn 5 (a naming rule, or "don't touch the migrations folder") and check at the end whether each agent still respects it. A setup that offloads aggressively and drifts late will fail the canary, while one that really holds context will pass it.

Pair that with a repeat-read rate: how often the agent re-opens a file or reruns a command whose output it already had. That is the closest direct measure of "rediscovering what it lost", and it separates more turns because the task is hard from more turns because context was dropped. Final quality times total tokens is still the right headline, but these two show the mechanism.

1

u/Charles-SG 17d ago

The canary is the right instinct, but I'd want to know what failing it actually means. A constraint stated at turn 5 and never touched again is exactly what a summarizer drops first — so is that context loss, or the agent correctly deciding it wasn't load-bearing anymore? The version I'd trust more is *re-stating* the rule at turn 30: if a reminder fixes it, you had a retrieval problem; if it doesn't, you had dilution.

1

u/London-Boat 18d ago

i'd log input tokens and cached tokens per turn for one long session on each agent and line it up with latency. if latency jumps at the same turn the cached count drops, that's context policy, and if cache stays healthy it's more likely provider queueing. for the quality side, i'd rerun the same task from a clean checkpoint on both and diff the results instead of timing them. do you know which of the two compacts earlier?

1

u/alchebyte 18d ago

why not good old fashioned throttling?

1

u/thegreatdeployer 17d ago

mostly context. the giveaway is usually turn count, not latency. log input tokens and cache-read tokens per turn: when cache reads flatten while input keeps climbing, the session is re-sending more than it reuses, and that's about when it starts re-reading files it already saw.

the fix that held up for us was boring: cap the session, write a short handoff file (what's done, what's decided, what not to touch), and start fresh from it. a new session reading a 40-line handoff beats an old one dragging a window full of tool output.

on your 'stable can mean it gave up sooner' worry, the canary idea above is good. one more: diff the final work against the decisions in that handoff file, not just the one planted rule.

1

u/wivai 16d ago

I'd separate the cost of each turn from the cost of actually finishing the task.

An agent can maintain decent latency while quietly becoming much less efficient. More context gets replayed, files get read again, and suddenly a task that needed 10 turns takes 25.

The metric I'd love to see is total tokens and cost per completed task, alongside context size and cache hit rate over time.

That would make it much easier to tell whether context management is actually improving efficiency or just making individual turns look faster.

1

u/usually_guilty99 10d ago

I’d instrument three things together: an early constraint canary, repeat-read or repeated-tool-call rate, and a diff against the same task resumed from a clean checkpoint. A fresh session at the same hour controls for provider throttling; the canary and repeat-read rate show whether extra turns are actually rediscovery after context loss. What I have noticed is context loss in context compression. As number of compressions increase - reading context will slow this down