arxiv.org·2026/10/22
分解式LLM服务把prefill与decode拆到不同节点池，KV缓存的计算与消费之间隔着网络，于是未命中不再只是本地重算，还得把缓存传给decode节点，而准入全局缓存同样要额外传输，糟糕的保留决策便同时浪费内存与带宽。作者为此提出KVLearn：轻量的前缀复用预测器（PRP）不改模型权重，用结构与时间特征估计复用概率；成本感知保留评分（CARS）按重算、传输与存储成本把概率转成保留/准入决策；自适应阈值控制器（ATC）依据实测命中率与内存压力在线调节准入阈值。框架集成进全局分解式服务拓扑，在文本与多模态负载（含图像/视频派生的大KV块）上评测；作者报告端到端首Token时延较No-Cache最多降56%，较LRU-Pool与Mooncake式基线分别降38%和33%，节点间KV传输量较LRU-Pool最多降53%，MM-Session上吞吐量与oracle相差约5%；代码开源。
为什么重要 解耦集群里 KV 保留决策同时决定显存与跨网带宽，直接吃掉 TTFT 与传输量；KVLearn 给出可学习闭环且已开源，正中推理基础设施方向。适合 做解耦推理 serving、KV cache 管理与集群性能的工程师
分离式（Disaggregated）LLM 服务将 prefill 与 decode 分离到不同节点池，并在 KV 缓存的计算与消费之间引入网络互联。这打破了经典缓存策略的一个关键假设：缓存未命中不再只带来本地重计算开销，还会引发到 decode 节点的网络传输。同样，将一个块接纳进全局缓存也需要额外传输，因此糟糕的保留决策会同时浪费内存和带宽。
我们提出 KVLearn，一个面向分离式服务的、基于学习的成本感知 KV 缓存保留框架。KVLearn 包含：(i) 轻量级的 Prefix Reuse Predictor（PRP），从结构与时间维度的前缀特征估计复用概率，无需修改模型权重；(ii) Cost-Aware Retention Score（CARS），利用重计算、传输与存储成本将复用概率转化为保留/接纳决策；(iii) Adaptive Threshold Controller（ATC），根据观测到的命中率和内存压力在线调整接纳阈值。
我们将 KVLearn 集成到全局分离式服务拓扑中，并在文本与多模态负载上评估，包括具有异构复用模式的大规模图像/视频派生 KV 块。KVLearn 将端到端 time-to-first-token 相比 No-Cache 降低最高 56%，相比 LRU-Pool 降低 38%，相比 Mooncake 式基线降低 33%，同时相比 LRU-Pool 将节点间 KV 传输量削减最高 53%。在 MM-Session 上，吞吐量保持在 oracle 的 ~5% 以内。代码已在 https://github.com/FastLM/KVLearn 开源。
Disaggregated textscLLM serving separates prefill and decode across node pools, placing a network fabric between textscKV-cache computation and consumption.
This breaks a key assumption of classical cache policies: a miss no longer incurs only local recomputation, but also network transfer to the decode node.
Likewise, admitting a block to the global cache requires an additional transfer, so poor retention decisions waste both memory and bandwidth.
We present KVLearn, a learning-based framework for cost-aware textscKV-cache retention in disaggregated serving.
KVLearn includes: (i) a lightweight Prefix Reuse Predictor (textscPRP) that estimates reuse probability from structural and temporal prefix features without modifying model weights; (ii) a Cost-Aware Retention Score (textscCARS) that converts reuse probability into keep/admit decisions using recomputation, transfer, and storage costs; and (iii) an Adaptive Threshold Controller (textscATC) that adjusts admission thresholds online from observed hit rates and memory pressure.
We integrate KVLearn into a globally disaggregated serving topology and evaluate it on text and multimodal workloads, including large image/video-derived textscKV blocks with heterogeneous reuse patterns.
KVLearn reduces end-to-end time-to-first-token by up to 56% versus No-Cache, 38% versus LRU-Pool, and 33% versus Mooncake-style baselines, while cutting inter-node textscKV transfer volume by up to 53% versus LRU-Pool.
On MM-Session, throughput remains within ∼5 of oracle. Code is available at https://github.com/FastLM/KVLearn.
注释 · 均为 KVLearn 相对下列基线的 up to 官方值；全文未获取，实验细节待核。出自 Abstract·结果段
EVIDENCE / METHODS
原文引语
解耦后 一次 miss = 本地重算 + 跨网传输到 decode 节点
a miss no longer incurs only local recomputation, but also network transfer to the decode node.
注释 · 准入全局缓存同样触发一次额外传输，坏决策同时浪费显存与带宽。出自 Abstract
原版排版与图表 · 手机无法内嵌时可在浏览器打开。PDF 插件内选文不能传给网页助手。