Humans use all of their senses to accomplish different tasks in everyday activities. In contrast, existing work on robotic manipulation mostly relies on one, or occasionally two modalities, such as vision and touch. In this work, we systematically study how visual, auditory, and tactile perception can jointly help robots to solve complex manipulation tasks. We build a robot system that can see with a camera, hear with a contact microphone, and feel with a vision-based tactile sensor, with all three sensory modalities fused with a self-attention model. Results on two challenging tasks, dense packing and pouring, demonstrate the necessity and power of multisensory perception for robotic manipulation: vision displays the global status of the robot but can often suffer from occlusion, audio provides immediate feedback of key moments that are even invisible, and touch offers precise local geometry for decision making. Leveraging all three modalities, our robotic system significantly outperforms prior methods.

Solving real-world sequential manipulation tasks requires robots to have a repertoire of skills applicable to a wide range of circumstances. To acquire such skills using data-driven approaches, we need massive and diverse training data which is often labor-intensive and non-trivial to collect and curate. In this work, we introduce Active Task Randomization (ATR), an approach that learns visuomotor skills for sequential manipulation by automatically creating feasible and novel tasks in simulation. During training, our approach procedurally generates tasks using a graph-based task parameterization. To adaptively estimate the feasibility and novelty of sampled tasks, we develop a relational neural network that maps each task parameter into a compact embedding. We demonstrate that our approach can automatically create suitable tasks for efficiently training the skill policies to handle diverse scenarios with a variety of objects. We evaluate our method on simulated and real-world sequential manipulation tasks by composing the learned skills using a task planner. Compared to baseline methods, the skills learned using our approach consistently achieve better success rates.

We present a retrospective on the state of Embodied AI research. Our analysis focuses on 13 challenges presented at the Embodied AI Workshop at CVPR. These challenges are grouped into three themes: (1) visual navigation, (2) rearrangement, and (3) embodied vision-and-language. We discuss the dominant datasets within each theme, evaluation metrics for the challenges, and the performance of state-of-the-art models. We highlight commonalities between top approaches to the challenges and identify potential future directions for Embodied AI research.

Modern multi-agent reinforcement learning frameworks rely on centralized training and reward shaping to perform well. However, centralized training and dense rewards are not readily available in the real world. Current multi-agent algorithms struggle to learn in the alternative setup of decentralized training or sparse rewards. To address these issues, we propose a self-supervised intrinsic reward ELIGN - expectation alignment - inspired by the self-organization principle in Zoology. Similar to how animals collaborate in a decentralized manner with those in their vicinity, agents trained with expectation alignment learn behaviors that match their neighbors' expectations. This allows the agents to learn collaborative behaviors without any external reward or centralized training. We demonstrate the efficacy of our approach across 6 tasks in the multi-agent particle and the complex Google Research football environments, comparing ELIGN to sparse and curiosity-based intrinsic rewards. When the number of agents increases, ELIGN scales well in all multi-agent tasks except for one where agents have different capabilities. We show that agent coordination improves through expectation alignment because agents learn to divide tasks amongst themselves, break coordination symmetries, and confuse adversaries. These results identify tasks where expectation alignment is a more useful strategy than curiosity-driven exploration for multi-agent coordination, enabling agents to do zero-shot coordination.

帕金森氏病（PD）是一种神经系统疾病，具有各种可观察到的与运动相关的症状，例如运动缓慢，震颤，肌肉僵硬和姿势受损。 PD通常通过评估运动障碍系统（例如运动障碍协会统一帕金森氏病评级量表（MDS-UPDRS））的评分系统来诊断PD。使用个体视频记录的自动严重性预测为无侵入性监测运动障碍提供了有希望的途径。但是，PD步态数据的大小有限阻碍模型能力和临床潜力。由于这种临床数据的稀缺性，并受到自我监督的大规模语言模型（例如GPT-3）的最新进展的启发，我们将人类运动预测用作有效的自我监督预训练的任务来估计运动障碍的严重性。我们介绍步态预测和损伤估计变压器，该变压器首先在公共数据集中进行预测以预测步态运动，然后应用于临床数据以预测MDS-UPDRS步态障碍的严重性。我们的方法的表现优于以前的方法，这些方法仅依赖于临床数据，从而达到了0.76的F1得分，精度为0.79，召回率为0.75。使用GaitForemer，我们展示了公共人类运动数据存储库如何通过学习通用运动表示来帮助临床用例。该代码可从https://github.com/markendo/gaitforemer获得。

预测以过去观察和电动机命令为条件的未来视觉观察的能力可以使体现的代理能够计划复杂环境中各种任务的解决方案。这项工作表明，我们可以通过掩盖的视觉建模预训练变压器来创建良好的视频预测模型。我们的方法名为MaskVit，基于两个简单的设计决策。首先，为了记忆和训练效率，我们使用两种类型的窗户注意力：时空和时空。其次，在训练期间，我们掩盖了一个可变百分比的令牌，而不是固定蒙版比率。对于推断，MaskVit通过迭代改进生成所有令牌，在该迭代中，我们会在掩码调度函数后逐步降低掩蔽率。在几个数据集上，我们证明了MaskVit优于视频预测中的先前作品，这是参数有效的，并且可以生成高分辨率视频（256x256）。此外，我们通过使用MaskVit在真实机器人上进行计划，证明了推理加速器的好处（最高512x）。我们的工作表明，我们可以通过利用最小的域知识的掩盖视觉建模的一般框架来赋予体现的代理具有强大的预测模型。

机器人在仓库和工厂等受控环境中执行重复和精确的敏感任务方面表现出色，但尚未扩展到体现在家庭任务中提供帮助的AI代理。受到基准在AI领域（例如计算机视觉和自然语言处理）中的催化效果的启发，社区正在寻找用于体现AI的新基准。体现AI基准的先前工作使用不同的形式主义定义任务，通常特定于一个环境，模拟器或域，从而难以开发一般和可比较的解决方案。在这项工作中，我们将一部分行为活动带入了栖息地2.0中，以从其快速模拟速度中受益，这是证明逻辑空间中定义的适应活动的第一步，将其定义为不同的模拟器。

数码相机的加速使用引起了人们对隐私和安全性的日益关注，尤其是在诸如行动识别之类的应用程序中。在本文中，我们提出了一个优化框架，以沿着人类行动识别管道提供强大的视觉隐私保护。我们的框架参数化了相机镜头，以成功地降低视频的质量，以抑制隐私属性并防止对抗性攻击，同时保持相关功能以进行活动识别。我们通过广泛的模拟和硬件实验来验证我们的方法。

在移动操作（MM）中，机器人可以在内部导航并与其环境进行交互，因此能够完成比仅能够导航或操纵的机器人的更多任务。在这项工作中，我们探讨如何应用模仿学习（IL）来学习MM任务的连续Visuo-Motor策略。许多事先工作表明，IL可以为操作或导航域训练Visuo-Motor策略，但很少有效应用IL到MM域。这样做是挑战的两个原因：在数据方面，当前的接口使得收集高质量的人类示范困难，在学习方面，有限数据培训的政策可能会在部署时遭受协变速转变。为了解决这些问题，我们首先提出了移动操作Roboturk（Momart），这是一种新颖的遥控框架，允许同时导航和操纵移动操纵器，并在现实的模拟厨房设置中收集一类大规模的大规模数据集。然后，我们提出了一个学习错误检测系统来解决通过检测代理处于潜在故障状态时的协变量转变。我们从该数据中培训表演者的IL政策和错误探测器，在专家数据培训时，在多个多级任务中达到超过45％的任务成功率和85％的错误检测成功率。 CodeBase，DataSets，Visualization，以及更多可用的https://sites.google.com/view/il-for-mm/home。

在过去的十年中，计算机愿景，旨在了解视觉世界的人工智能分支，从简单地识别图像中的物体来描述图片，回答有关图像的问题，以及围绕物理空间的机器人操纵甚至产生新的视觉内容。随着这些任务和应用程序的现代化，因此依赖更多数据，用于模型培训或评估。在本章中，我们展示了新颖的互动策略可以为计算机愿景提供新的数据收集和评估。首先，我们提出了一种众群界面，以通过数量级加速付费数据收集，喂养现代视觉模型的数据饥饿性质。其次，我们探索使用自动社交干预措施增加志愿者贡献的方法。第三，我们开发一个系统，以确保人类对生成视觉模型的评估是可靠的，实惠和接地在心理物理学理论中。我们结束了人机互动的未来机会，以帮助计算机愿景。