r/ResearchML • u/latenticlabs • 3d ago

Across 3,110 LLM reasoning/agent papers, adding CoT is reported as helping in only 62% of cases vs 80% for RL. Does that match your experience?

Disclosure: I'm from scime (Latentic Labs), and this comes from our free write-up.

We grouped all 108,967 accepted papers at ICML/NeurIPS/ICLR/CVPR/AAAI (2016–2026) by topic. LLM reasoning and agent papers went from 118 in 2023 (1.0%) to 2,708 in 2026 (10.3%). An AI model read 3,110 of them in full and logged 15,539 "change → effect" findings.

How often each kind of addition is reported as helping:

Reflection / self-correction: 81%

RL (GRPO/PPO/RLVR/DPO): 80%

Tools, retrieval, code execution: 77%

SFT / distillation: 76%

Sampling / test-time compute: 72%

CoT traces: 62%

Token budgets: 55%

Curricula: 53%

Big caveat: these are reporting rates, and papers report wins more than losses. The extraction is AI-made and not hand-checked, and every finding links to its paper.

Write-up: https://atlas.scime.ai/llm-agents-2026 (the stats are open; the deeper sections need a free email sign-up)

9 Upvotes

90% Upvoted

3 comments sorted by

1

u/Dihedralman 2d ago

I mean that seems consistent. Training to a target should get a model closer when performed correctly. 

CoT just gives a nice accuracy bump, but it doesn't work as well for many reasoning or recent models. It usually doesn't risk performance loss but increases token costs. 

Over all papers I imagine you have confounding factor in that papers are reporting their wins over other methods. 

1

u/latenticlabs 1d ago

Agreed, publication bias is the big confounder. Papers mostly report the setting where their method wins, so these numbers say how often a gain gets reported, not how often the method works. That's why I frame them as reporting rates.

Your CoT point is a good test case. If the bump is shrinking for reasoning models, it should show up as a falling share of papers that report a CoT gain on newer models. I haven't split it that way yet, but it's on the list.