Quantifying motion in 3D is important for studying the behavior of humans and other animals, but manual pose annotations are expensive and time-consuming to obtain. Self-supervised keypoint discovery is a promising strategy for estimating 3D poses without annotations. However, current keypoint discovery approaches commonly process single 2D views and do not operate in the 3D space. We propose a new method to perform self-supervised keypoint discovery in 3D from multi-view videos of behaving agents, without any keypoint or bounding box supervision in 2D or 3D. Our method uses an encoder-decoder architecture with a 3D volumetric heatmap, trained to reconstruct spatiotemporal differences across multiple views, in addition to joint length constraints on a learned 3D skeleton of the subject. In this way, we discover keypoints without requiring manual supervision in videos of humans and rats, demonstrating the potential of 3D keypoint discovery for studying behavior.

我们提出了一种学习来自未标识的行为视频的代理的姿势和结构的方法。从观察开始，表现代理通常是行为视频中的主要运动来源，我们的方法使用具有几何瓶颈的编码器 - 解码器架构来重建视频帧之间的差异。只要仅关注运动区域，我们的方法直接在输入视频上工作，而无需手动注释，例如关键点或边界框。关于各种代理类型（鼠标，飞，人，水母和树木）的实验展示了我们的方法的一般性，并揭示了我们发现的关键点代表着语义有意义的身体部位，这在关键点回归上实现了最先进的性能在自我监督的方法中。此外，我们发现的关键点可实现可比的性能，以对下游任务的监督关键点，例如行为分类，表明我们的方法可以大大降低模型培训VIS-VIS监督方法的成本。

诸如关键点之类的结构化表示形式被广泛用于姿势传输，条件图像生成，动画和3D重建。但是，他们的监督学习需要每个目标域的昂贵注释。我们提出了一种自我监督的方法，该方法学会从外观上脱离对象结构，并用直边链接的2D关键点的图形。只有描绘同一对象类的图像集合，都学会了关键点的位置及其成对边缘权重。该图是可以解释的，例如，当应用于显示人的图像时，自动链接会恢复人类骨架拓扑。我们的关键要素是i）一个编码器，该编码器可预测输入图像中的关键点位置，ii）共享图作为一个潜在变量，该图形在每个图像中链接了相同的对键点，iii）一个中间边缘映射，结合了潜在图形边缘权重和关键点的位置以柔软，可区分的方式以及iv）在随机掩盖的图像上的介入目标。尽管更简单，但自动链接在已建立的关键点上优于现有的自我监督方法，并构成估计基准，并为更多样化的数据集上的结构调节生成模型铺平了道路。

通过手动创建或使用3D扫描工具来创建高质量的铰接3D动物3D模型。因此，从2D图像重建铰接的3D对象的技术至关重要且非常有用。在这项工作中，我们提出了一个实用问题设置，以估算只有几个（10-30）特定动物物种（例如马）的野外图像（Horse）的3D姿势和形状。与依赖于预定义模板形状的现有作品相反，我们不假设任何形式的2D或3D地面真相注释，也不利用任何多视图或时间信息。此外，每个输入图像合奏都可以包含具有不同姿势，背景，照明和纹理的动物实例。我们的主要见解是，与整体动物相比，3D零件的形状要简单得多，并且它们是强大的W.R.T.动物姿势关节。遵循这些见解，我们提出了Lassie，这是一个新颖的优化框架，以最少的用户干预以自我监督的方式发现3D部分。 Lassie背后的关键推动力是使用自我篇幅的深度功能实现2D-3D零件的一致性。与先前的艺术相比，关于Pascal-Part和自我收集的野生动物数据集的实验表明，3D重建以及2D和3D部分的发现都更好。项目页面：chhankyo.github.io/lassie/

Its numerous applications make multi-human 3D pose estimation a remarkably impactful area of research. Nevertheless, assuming a multiple-view system composed of several regular RGB cameras, 3D multi-pose estimation presents several challenges. First of all, each person must be uniquely identified in the different views to separate the 2D information provided by the cameras. Secondly, the 3D pose estimation process from the multi-view 2D information of each person must be robust against noise and potential occlusions in the scenario. In this work, we address these two challenges with the help of deep learning. Specifically, we present a model based on Graph Neural Networks capable of predicting the cross-view correspondence of the people in the scenario along with a Multilayer Perceptron that takes the 2D points to yield the 3D poses of each person. These two models are trained in a self-supervised manner, thus avoiding the need for large datasets with 3D annotations.

我们建议在不使用任何手动注释的情况下为人类的姿势估算器进行个性化。虽然人类姿势估计有重大进步，但对于模型来说概括到不同的未知环境和看不见的人仍然非常具有挑战性。我们而不是为每个测试用例使用固定模型，我们在测试时间期间调整我们的姿势估算器以利用特定人的信息。我们首先将我们的模型与监督和自我监督的姿势估算目标相同。我们使用变压器模型来构建自我监控的关键点和监督关键点之间的转换。在测试时间期间，我们通过通过自我监督目标进行微调来个性化和调整我们的模型。然后通过转换更新的自我监督的关键点来改进姿势。我们尝试多个数据集，并显示我们自我监督个性化的姿势估算的显着改进。

To date, little attention has been given to multi-view 3D human mesh estimation, despite real-life applicability (e.g., motion capture, sport analysis) and robustness to single-view ambiguities. Existing solutions typically suffer from poor generalization performance to new settings, largely due to the limited diversity of image-mesh pairs in multi-view training data. To address this shortcoming, people have explored the use of synthetic images. But besides the usual impact of visual gap between rendered and target data, synthetic-data-driven multi-view estimators also suffer from overfitting to the camera viewpoint distribution sampled during training which usually differs from real-world distributions. Tackling both challenges, we propose a novel simulation-based training pipeline for multi-view human mesh recovery, which (a) relies on intermediate 2D representations which are more robust to synthetic-to-real domain gap; (b) leverages learnable calibration and triangulation to adapt to more diversified camera setups; and (c) progressively aggregates multi-view information in a canonical 3D space to remove ambiguities in 2D representations. Through extensive benchmarking, we demonstrate the superiority of the proposed solution especially for unseen in-the-wild scenarios.

Recently developed methods for video analysis, especially models for pose estimation and behavior classification, are transforming behavioral quantification to be more precise, scalable, and reproducible in fields such as neuroscience and ethology. These tools overcome long-standing limitations of manual scoring of video frames and traditional "center of mass" tracking algorithms to enable video analysis at scale. The expansion of open-source tools for video acquisition and analysis has led to new experimental approaches to understand behavior. Here, we review currently available open-source tools for video analysis and discuss how to set up these methods for labs new to video recording. We also discuss best practices for developing and using video analysis methods, including community-wide standards and critical needs for the open sharing of datasets and code, more widespread comparisons of video analysis methods, and better documentation for these methods especially for new users. We encourage broader adoption and continued development of these tools, which have tremendous potential for accelerating scientific progress in understanding the brain and behavior.

In this work, we demonstrate that 3D poses in video can be effectively estimated with a fully convolutional model based on dilated temporal convolutions over 2D keypoints. We also introduce back-projection, a simple and effective semi-supervised training method that leverages unlabeled video data. We start with predicted 2D keypoints for unlabeled video, then estimate 3D poses and finally back-project to the input 2D keypoints. In the supervised setting, our fully-convolutional model outperforms the previous best result from the literature by 6 mm mean per-joint position error on Human3.6M, corresponding to an error reduction of 11%, and the model also shows significant improvements on HumanEva-I. Moreover, experiments with back-projection show that it comfortably outperforms previous state-of-the-art results in semisupervised settings where labeled data is scarce. Code and models are available at https://github.com/ facebookresearch/VideoPose3D

在深度学习的时代，具有未知校准未知校准的多个摄像机的人类姿态估计几乎没有关注迄今为止。我们展示如何培训一个神经模型，以高精度和最小延迟开销来执行此任务。由于多视图闭塞，所提出的模型考虑了联合位置不确定性，并且只需要2D关键点数据进行培训。我们的方法优于良好的人机3.6M数据集上的经典捆绑调整和弱监督单眼3D基线，以及野外滑雪姿势PTZ数据集的更具挑战性。

人类性能捕获是一种非常重要的计算机视觉问题，在电影制作和虚拟/增强现实中具有许多应用。许多以前的性能捕获方法需要昂贵的多视图设置，或者没有恢复具有帧到帧对应关系的密集时空相干几何。我们提出了一种新颖的深度致密人体性能捕获的深层学习方法。我们的方法是基于多视图监督的弱监督方式培训，完全删除了使用3D地面真理注释的培训数据的需求。网络架构基于两个单独的网络，将任务解散为姿势估计和非刚性表面变形步骤。广泛的定性和定量评估表明，我们的方法在质量和稳健性方面优于现有技术。这项工作是DeepCAP的扩展版本，在那里我们提供更详细的解释，比较和结果以及应用程序。

大多数实时人类姿势估计方法都基于检测接头位置。使用检测到的关节位置，可以计算偏差和肢体的俯仰。然而，由于这种旋转轴仍然不观察，因此不能计算沿着肢体沿着肢体至关重要的曲折，这对于诸如体育分析和计算机动画至关重要。在本文中，我们引入了方向关键点，一种用于估计骨骼关节的全位置和旋转的新方法，仅使用单帧RGB图像。灵感来自Motion-Capture Systems如何使用一组点标记来估计全骨骼旋转，我们的方法使用虚拟标记来生成足够的信息，以便准确地推断使用简单的后处理。旋转预测改善了接头角度最佳报告的平均误差48％，并且在15个骨骼旋转中实现了93％的精度。该方法还通过MPJPE在原理数据集上测量，通过MPJPE测量，该方法还改善了当前的最新结果14％，并概括为野外数据集。

Deep learning-based 3D human pose estimation performs best when trained on large amounts of labeled data, making combined learning from many datasets an important research direction. One obstacle to this endeavor are the different skeleton formats provided by different datasets, i.e., they do not label the same set of anatomical landmarks. There is little prior research on how to best supervise one model with such discrepant labels. We show that simply using separate output heads for different skeletons results in inconsistent depth estimates and insufficient information sharing across skeletons. As a remedy, we propose a novel affine-combining autoencoder (ACAE) method to perform dimensionality reduction on the number of landmarks. The discovered latent 3D points capture the redundancy among skeletons, enabling enhanced information sharing when used for consistency regularization. Our approach scales to an extreme multi-dataset regime, where we use 28 3D human pose datasets to supervise one model, which outperforms prior work on a range of benchmarks, including the challenging 3D Poses in the Wild (3DPW) dataset. Our code and models are available for research purposes.

3D human whole-body pose estimation aims to localize precise 3D keypoints on the entire human body, including the face, hands, body, and feet. Due to the lack of a large-scale fully annotated 3D whole-body dataset, a common approach has been to train several deep networks separately on datasets dedicated to specific body parts, and combine them during inference. This approach suffers from complex training and inference pipelines because of the different biases in each dataset used. It also lacks a common benchmark which makes it difficult to compare different methods. To address these issues, we introduce Human3.6M 3D WholeBody (H3WB) which provides whole-body annotations for the Human3.6M dataset using the COCO Wholebody layout. H3WB is a large scale dataset with 133 whole-body keypoint annotations on 100K images, made possible by our new multi-view pipeline. Along with H3WB, we propose 3 tasks: i) 3D whole-body pose lifting from 2D complete whole-body pose, ii) 3D whole-body pose lifting from 2D incomplete whole-body pose, iii) 3D whole-body pose estimation from a single RGB image. We also report several baselines from popular methods for these tasks. The dataset is publicly available at \url{https://github.com/wholebody3d/wholebody3d}.

当标记的数据丰富时，从单个图像中进行3D姿势估计的监督方法非常有效。但是，由于对地面3D标签的获取是劳动密集型且耗时的，最近的关注已转向半决赛和弱监督的学习。产生有效的监督形式，几乎没有注释，仍然在拥挤的场景中构成重大挑战。在本文中，我们建议通过加权区分三角剖分施加多视文几何约束，并在没有标签时将其用作一种自我设计的形式。因此，我们以一种方式训练2D姿势估计器，以使其预测对应于对三角姿势的3D姿势的重新投影，并在其上训练辅助网络以产生最终的3D姿势。我们通过一种加权机制来补充三角剖分，从而减轻了由自我咬合或其他受试者的遮挡引起的嘈杂预测的影响。我们证明了半监督方法对人类36M和MPI-INF-3DHP数据集的有效性，以及在具有闭塞的新的多视频多人数据集上。

Understanding the 3D world without supervision is currently a major challenge in computer vision as the annotations required to supervise deep networks for tasks in this domain are expensive to obtain on a large scale. In this paper, we address the problem of unsupervised viewpoint estimation. We formulate this as a self-supervised learning task, where image reconstruction provides the supervision needed to predict the camera viewpoint. Specifically, we make use of pairs of images of the same object at training time, from unknown viewpoints, to self-supervise training by combining the viewpoint information from one image with the appearance information from the other. We demonstrate that using a perspective spatial transformer allows efficient viewpoint learning, outperforming existing unsupervised approaches on synthetic data, and obtains competitive results on the challenging PASCAL3D+ dataset.

全面监督的人类网格恢复方法是渴望数据的，由于3D规定基准数据集的可用性有限和多样性，因此具有较差的概括性。使用合成数据驱动的训练范例，已经从合成配对的2D表示（例如2D关键点和分段掩码）和3D网格中训练了模型的最新进展，其中已使用合成数据驱动的训练范例和3D网格进行了训练。但是，由于合成训练数据和实际测试数据之间的域间隙很难解决2D密集表示，因此很少探索合成密集的对应图（即IUV）。为了减轻IUV上的这个领域差距，我们提出了使用可靠但稀疏表示的互补信息（2D关键点）提出的交叉代理对齐。具体而言，初始网格估计和两个2D表示之间的比对误差将转发为回归器，并在以下网格回归中动态校正。这种适应性的交叉代理对准明确地从偏差和捕获互补信息中学习：从稀疏的表示和浓郁的浓度中的稳健性。我们对多个标准基准数据集进行了广泛的实验，并展示了竞争结果，帮助减少在人类网格估计中生产最新模型所需的注释工作。

多个摄像机制造的视频录制的可用性越来越多，为姿势和运动重建方法中的减少和深度歧义提供了新的方法。然而，多视图算法强烈依赖于相机参数;特别地，相机之间的相对介绍。在不受控制的设置中，这种依赖变为一旦转移到动态捕获一次。我们介绍Flex（免费多视图重建），一个端到端的无参数多视图模型。 Flex是无意义的参数，即它不需要任何相机参数，都不是内在的也不是外在的。我们的关键思想是骨架部件和骨长之间的3D角度是不变的相机位置。因此，学习3D旋转和骨长而不是位置允许预测所有相机视图的公共值。我们的网络采用多个视频流，学习通过新型多视图融合层的融合深度特征，并重建单一一致的骨架，其具有时间上相干的关节旋转。我们展示了人类3.6M和KTH多视图足球II数据集的定量和定性结果，以及动态摄像头捕获的合成多人视频流。我们将模型与最先进的方法进行比较，这些方法没有参与参数，并在没有相机参数的情况下显示，我们在获得相机参数可用时获取可比结果的同时优于较大的余量。我们的项目页面上可以使用代码，培训的模型，视频示例和更多材料。

对人类姿势和行动的认可对于自治系统与人们顺利互动。然而，相机通常在2D中捕获人类的姿势，作为图像和视频，这在跨越识别任务具有挑战性的观点来具有显着的外观变化。为了解决这个问题，我们探讨了来自2D信息的3D人体姿势中的识别相似性，在现有工作中没有得到很好地研究。在这里，我们提出了一种从2D主体关节键盘学习紧凑型视图 - 不变的嵌入空间的方法，而不明确地预测3D姿势。通过确定性映射难以代表预测和遮挡的2D姿势的输入模糊，因此我们采用了嵌入空间的概率制定。实验结果表明，与3D姿态估计模型相比，我们的嵌入模型在不同相机视图中检索类似的姿势时达到更高的准确性。我们还表明，通过培训简单的时间嵌入模型，我们在姿势序列检索方面取得了卓越的性能，并大大减少了基于堆叠帧的嵌入式的嵌入维度，以实现高效的大规模检索。此外，为了使我们的嵌入能够使用部分可见的输入，我们进一步调查培训期间的不同关键点遮挡增强策略。我们证明这些遮挡增强显着提高了部分2D输入姿势的检索性能。行动识别和视频对齐的结果表明，使用我们的嵌入没有任何额外培训，可以实现相对于每个任务专门培训的其他模型的竞争性能。

尽管最近的进步，但是，尽管最近的进展，但是从单个图像中的人类姿势的全3D估计仍然是一个具有挑战性的任务。在本文中，我们探讨了关于场景几何体的强先前信息的假设可用于提高姿态估计精度。为了主弱地解决这个问题，我们已经组装了一种新的$ \ textbf {几何姿势提供} $ DataSet，包括与各种丰富的3D环境交互的人员的多视图图像。我们利用商业运动捕获系统来收集场景本身的姿势和构造精确的几何3D CAD模型的金标估计。要将对现有框架的现有框架注入图像的现有框架，我们介绍了一种新颖的，基于视图的场景几何形状，一个$ \ textbf {多层深度图} $，它采用了多次射线跟踪到简明地编码沿着每种相机视图光线方向的多个表面入口和退出点。我们提出了两种不同的机制，用于集成多层深度信息姿势估计：输入作为升降2D姿势的编码光线特征，其次是促进学习模型以支持几何一致姿态估计的可差异损失。我们通过实验展示这些技术可以提高3D姿势估计的准确性，特别是在遮挡和复杂场景几何形状的存在中。