数据源与状态仍在导航菜单中
arxiv.org·2026/10/19
针对视觉难判接触力、机器人触觉数据稀缺的问题,Ego2Contact 从人类自我中心视频学习视觉-触觉先验并迁移到机器人手:以 DINOv2 全局场景 token 与 WiLoR 手部局部 token 经时序 transformer、交叉注意力融合,再用条件流匹配在规范双手网格上生成触觉场;解剖区域对应把不同机器人触觉布局映射到该网格,前向映射监督微调,反向映射输出 PPO 用的区域压力/接触标志或密集触觉图像供 VLA 后训练。人类数据上,EgoTouch 未见物体与 EgoTactile 跨数据集指标均最优,未见物体 C.IoU 达 0.4812,超过 TouchAnything;仿真适应后网格 C.IoU 升至 0.476。多项 Isaac Gym 任务中,Bottle Cap Turning 成功率 71.0%,高于无触觉基线 64.1% 并接近真触觉 74.1%,但 Pen Cap Removal 仍远低于真触觉。真实机器人上以遥操作数据适配预测器并对 Test Tube Transfer 与 Bottle Cap Removal 做策略后训练,作者报告改进非触觉基线并恢复部分真触觉收益;局限是仍需手工区域对应,未来拟降低对机器人数据的依赖。
为什么重要 让缺触觉硬件的机器人策略用 RGB 推断触觉,降低真机触觉数据采集与复用门槛适合 做机器人触觉、VLA 策略与跨本体数据闭环的工程师
接触位置和交互力对于富接触的机器人操作至关重要,但仅凭视觉往往难以分辨。在机器人上测量这些量需要专门的触觉硬件和针对特定平台的数据采集,这使得机器人触觉数据一直稀缺。人类第一人称视频的规模化要容易得多,但缺乏触觉。近期在配对触觉手套测量值的第一人称录像上训练的预测器,能够仅凭视频预测这一缺失信号。然而,这些预测器仍局限于人手,而人手在外观和触觉布局上与机器人手不同。
我们提出Ego2Contact,它从人类交互中学习视觉-触觉先验,并将其迁移到机器人手上。其预测器将时序场景上下文与手部局部视觉特征相融合,从RGB视频中推断双手触觉场。解剖区域对应关系将各不相同的机器人触觉布局与预测器的规范网格对齐,从而将机器人测量值转化为微调目标,将预测结果转化为策略输入。
Ego2Contact在人类数据(包括未见过的物体)上以及适配后的机器人数据上均优于先前的触觉预测器。适配后的预测器为仿真强化学习和真实机器人策略后训练提供触觉输入。在仿真中,使用预测触觉的策略优于非触觉基线,并恢复了真值触觉感知的部分收益。这些结果表明,当直接触觉感知受限时,从人类学习到的触觉先验可以提供触觉。
Contact locations and interaction forces are critical for contact-rich robot manipulation but are often ambiguous from vision alone.
Measuring them on robots requires specialized tactile hardware and platform-specific data collection, which keeps robot tactile data scarce.
Human egocentric video is far easier to scale but lacks touch.
Recent predictors trained on egocentric recordings paired with tactile-glove measurements can predict this missing signal from video alone.
These predictors, however, remain confined to human hands, which differ from robot hands in appearance and tactile layout.
We present Ego2Contact, which learns a visual-tactile prior from human interactions and transfers it to robot hands.
Its predictor fuses temporal scene context with hand-local visual features to infer bimanual tactile fields from the RGB video.
Anatomical region correspondences align varying robot tactile layouts with the predictor's canonical grid, turning robot measurements into fine-tuning targets and predictions into policy inputs.
Ego2Contact outperforms prior tactile predictors on human data, including unseen objects, and on robot data after adaptation.
The adapted predictor provides tactile inputs for both simulated reinforcement learning and real-robot policy post-training.
In simulation, policies using predicted touch outperform non-tactile baselines and recover part of the benefit of ground-truth tactile sensing.
These results show that human-learned tactile priors can supply touch when direct tactile sensing is limited.



EVIDENCE / METHODS
要点
接触位置与交互力对接触密集操作关键,但仅靠视觉常存在歧义。
原版排版与图表 · 手机无法内嵌时可在浏览器打开。PDF 插件内选文不能传给网页助手。