arxiv.org·2026/10/26
现有轨迹研究侧重任务无关行为模式,难解释任务理解错误导致的错误补丁;作者在 SWE-bench Pro 上用 mini-swe-agent 运行 DeepSeek 与 Qwen 系列开源推理模型,采集 400 条轨迹分析原生 CoT 中的 task-grounded reasoning。LLM 提取原子主张并归为 O/G/S/F,评估代理比对仓库与库文档证据找出 refuted 项,再手动追溯缺陷并经 claim-corrected replay 确认因果。这类推理仅占 CoT tokens 的 18.5%,至少 9.6% 主张被反驳;错误多误述现有代码行为,且绝大多数已解决轨迹也含 faulty claims,其存在并非失败强信号。未解决轨迹中 67.6% 追溯到 faulty claims,纠正一个关键 claim 在 89.0% 重放中修复关联缺陷,但仅少数解决任务。结论是 CoT 含任务特异诊断信号,须精准定位真正影响补丁的少数 faulty claims;研究仅覆盖开源模型、同一 scaffold 与基准,错误率为下界。
为什么重要 它给出从 native CoT 做失败归因与提交前修复的证据:错误声明常见但多数无害,必须定位 patch 真正依赖的 key faulty claim。适合 做 coding agent 评测、失败归因与运行时修复的工程师
基于大型语言模型(LLM)的编码智能体越来越多地解决现实世界的软件工程任务,但仍在许多任务上失败。先前的轨迹研究通过任务无关的智能体行为模式(如重复动作)来刻画这些失败。然而,智能体可能遵循了合理的工作流,却对现有代码、任务需求或候选修复产生了错误理解。驱动大多数最先进编码智能体的推理模型,在其原生思维链(CoT)中揭示了这种理解,其中关于任务的主张可以与事实进行核对。但仍不清楚 CoT 中有多少这样的可核对主张,有多少是错误的,以及错误主张是否真的导致错误补丁。
我们首次对这类主张进行实证研究,并称之为任务锚定推理(task-grounded reasoning),分析了四个开源权重推理模型在 SWE-bench Pro 上的 400 条轨迹。我们用 LLM 从 CoT 中提取原子主张,根据 LLM 智能体收集的证据评估它们,并手动将错误补丁的缺陷追溯到错误主张,通过修正主张后重放来确认每个环节。任务锚定推理仅覆盖 CoT token 的 18.5%,但每条轨迹产生 191 个原子主张,其中至少 9.6% 是错误的。错误主张还出现在 96.7% 的已解决轨迹中,因此它们仅出现是失败的弱信号。然而,188 个错误补丁中有三分之二可追溯到错误主张。在 89.0% 的情况下,修正这样一个主张会使智能体修复关联缺陷,而修正所有此类主张可解决 11.7% 的失败任务。
我们的发现表明,CoT 携带用于诊断和修复智能体失败的任务特定信号,但利用这些信号需要精确定位少数真正导致补丁缺陷的错误主张。
Large language model (LLM)-based coding agents increasingly resolve real-world software engineering tasks, yet still fail on many of them.
Prior trajectory studies characterize these failures through task-agnostic patterns of agent behavior, such as repeated actions.
However, an agent may follow a sound workflow yet act on a faulty understanding of the existing code, the task requirements, or a candidate fix.
Reasoning models, which power most state-of-the-art coding agents, reveal this understanding in their native chain-of-thought (CoT), whose claims about the task can be checked against the facts.
Yet it remains unclear how many such checkable claims the CoT contains, how many are faulty, and whether faulty claims actually cause incorrect patches.
We present the first empirical study of such claims, which we term task-grounded reasoning, analyzing 400 trajectories of four open-weight reasoning models on SWE-bench Pro.
We extract atomic claims from the CoT with an LLM, assess them against evidence gathered by an LLM agent, and manually trace the defects of incorrect patches back to faulty claims, confirming each link with a claim-corrected replay.
Task-grounded reasoning covers only 18.5% of the CoT tokens, yet yields 191 atomic claims per trajectory, at least 9.6% of which are faulty.
Faulty claims also occur in 96.7% of the resolved trajectories, so their mere presence is a weak signal of failure. Yet two thirds of the 188 incorrect patches trace back to faulty claims.
Correcting such a claim makes the agent fix the linked defect in 89.0% of the cases, and correcting all such claims resolves 11.7% of the failed tasks.
Our findings show that the CoT carries task-specific signals for diagnosing and repairing agent failures, but exploiting them requires pinpointing the few faulty claims that actually cause patch defects.
CoT 占比
任务声明覆盖 token 比例
声明密度
平均每条轨迹推理单元数
总步数
全部轨迹合计
推理单元
提取出的原子声明总数
注释 · TGR 指可依据任务说明、仓库与库文档判定的声明。出自 §3.1
注释 · S 可从任务语句读出,O 需推断代码,F 预测未写代码。出自 §3.2.2

EVIDENCE / METHODS
建议
提交前检查点 只纠正 patch 依赖的 术语解释与补丁缺陷直接相关、纠正后有机会消除该缺陷的错误声明。
落地方式不是把全部错误声明都报给 agent,而是在提交前验证最终 patch 仍依赖的声明:对代码行为用执行探针,对修复机制用目标测试,对需求与任务语句比对。评估 agent 通过 shell、官方文档抓取、citation、submission 工具取证;方法依赖 native CoT,隐藏思维链的商业模型不适用。
注释 · 优先检查最终 patch 依赖的声明;仅查标识符会漏掉行为误解。出自 §4.1
原版排版与图表 · 手机无法内嵌时可在浏览器打开。PDF 插件内选文不能传给网页助手。