arxiv.org·2026/10/26
为什么重要 对构建客服/运维助手有直接启示：基于重写为自助指导的解答做 QLoRA 微调最有效，但端到端编排未见增益，返回单工具答案反而更稳。
大多数到达 HPC 帮助台的问题已经在中心的用户指南中或上学期其他人提交的工单中得到解答；用户仍会提交新工单，因为指南难以导航，且工单历史不归他们搜索。我们提出一个从这两个来源回答的助手：一个编排器通过 Model Context Protocol 调用两个工具——对实时文档进行检索增强生成（RAG），以及一个在从历史工单挖掘出的 11,244 个查询-解答对上用 QLoRA 微调的 Llama-3-8B 模型——并合并它们的答案。由于工单数据不得离开机构，所有模型都在本地或大学托管的 LLM 服务上运行。
在来自 Purdue 的 Rosen Center for Advanced Computing 的 400 张留出工单上（与训练集不相交且无近重复），由来自未生成任何答案的模型家族的两名评审评估，基于改写为自助指导的解答进行微调在与 RCAC 解答的一致性上是明显赢家（BERTScore F1 从基础模型的 0.06 提升到 0.27），并在平均正确性上被评为最高。端到端系统并未超过其工具，尽管它们是互补的：对每个问题选择更好的答案会超过任一工具（判定正确性 3.41 对比 3.20，这是一个预言机上限），而合并答案在 78% 的回答中跟随文档工具。改为返回单一工具的答案可恢复微调工具的质量（3.19 对比 2.66），但无论是学习到的选择器还是 LLM 验证器都没有超过该工具本身。
Most questions that reach an HPC help desk have already been answered in the center's user guides or in a ticket someone else filed last semester; users still file new tickets because the guides are hard to navigate and the ticket history is not theirs to search.
We present an assistant that answers from both sources: an orchestrator calls two tools over the Model Context Protocol, retrieval-augmented generation (RAG) over the live documentation and a Llama-3-8B model fine-tuned with QLoRA on 11,244 query-resolution pairs mined from historical tickets, and merges their answers.
Every model runs on-premises or on the university's hosted LLM service because ticket data may not leave the institution.
On 400 held-out tickets from Purdue's Rosen Center for Advanced Computing, disjoint from training and free of near-duplicates, and evaluated by two judges from model families that generated none of the answers, fine-tuning on resolutions rewritten into self-service guidance is the clear winner in agreement with RCAC's resolutions (BERTScore F1 improves from 0.06 to 0.27 over the base model) and is judged most correct on average.
The end-to-end system does not improve on its tools, although they are complementary: choosing the better answer per question would exceed either tool (judged correctness 3.41 versus 3.20, an oracle bound), while the merge follows the documentation tool in 78% of answers.
Returning one tool's answer instead recovers the fine-tuned tool's quality (3.19 versus 2.66), but neither a learned selector nor an LLM verifier beats that tool alone.
原版排版与图表 · 手机无法内嵌时可在浏览器打开。PDF 插件内选文不能传给网页助手。