手工姿势和形状估计研究领域的数据集和工具的数量和质量作为所做的重大进展的证据。然而，即使是迄今为止报告的最高质量的数据集，也具有注释的缺点。我们提出了一种基于可分辨率的射线跟踪的细化方法，并演示了如何具有高质量的公共可用的，双摄像机数据集（Interwand2.6m）可以成为一个更好的数据集，相对于注释质量。到目前为止，迄今未采用可分辨率的射线跟踪，特此被证明优于过去已经采用的近似替代品。为了解决缺乏可靠的地面真理，就量化评估而言，我们求助于现实的合成数据，表明我们诱导的改进确实很重要。通过视觉评估，实际数据中的实际数据也是如此。

We present HARP (HAnd Reconstruction and Personalization), a personalized hand avatar creation approach that takes a short monocular RGB video of a human hand as input and reconstructs a faithful hand avatar exhibiting a high-fidelity appearance and geometry. In contrast to the major trend of neural implicit representations, HARP models a hand with a mesh-based parametric hand model, a vertex displacement map, a normal map, and an albedo without any neural components. As validated by our experiments, the explicit nature of our representation enables a truly scalable, robust, and efficient approach to hand avatar creation. HARP is optimized via gradient descent from a short sequence captured by a hand-held mobile phone and can be directly used in AR/VR applications with real-time rendering capability. To enable this, we carefully design and implement a shadow-aware differentiable rendering scheme that is robust to high degree articulations and self-shadowing regularly present in hand motion sequences, as well as challenging lighting conditions. It also generalizes to unseen poses and novel viewpoints, producing photo-realistic renderings of hand animations performing highly-articulated motions. Furthermore, the learned HARP representation can be used for improving 3D hand pose estimation quality in challenging viewpoints. The key advantages of HARP are validated by the in-depth analyses on appearance reconstruction, novel-view and novel pose synthesis, and 3D hand pose refinement. It is an AR/VR-ready personalized hand representation that shows superior fidelity and scalability.

We introduce Structured 3D Features, a model based on a novel implicit 3D representation that pools pixel-aligned image features onto dense 3D points sampled from a parametric, statistical human mesh surface. The 3D points have associated semantics and can move freely in 3D space. This allows for optimal coverage of the person of interest, beyond just the body shape, which in turn, additionally helps modeling accessories, hair, and loose clothing. Owing to this, we present a complete 3D transformer-based attention framework which, given a single image of a person in an unconstrained pose, generates an animatable 3D reconstruction with albedo and illumination decomposition, as a result of a single end-to-end model, trained semi-supervised, and with no additional postprocessing. We show that our S3F model surpasses the previous state-of-the-art on various tasks, including monocular 3D reconstruction, as well as albedo and shading estimation. Moreover, we show that the proposed methodology allows novel view synthesis, relighting, and re-posing the reconstruction, and can naturally be extended to handle multiple input images (e.g. different views of a person, or the same view, in different poses, in video). Finally, we demonstrate the editing capabilities of our model for 3D virtual try-on applications.

我们提出了一种称为DPODV2（密集姿势对象检测器）的三个阶段6 DOF对象检测方法，该方法依赖于致密的对应关系。我们将2D对象检测器与密集的对应关系网络和多视图姿势细化方法相结合，以估计完整的6 DOF姿势。与通常仅限于单眼RGB图像的其他深度学习方法不同，我们提出了一个统一的深度学习网络，允许使用不同的成像方式（RGB或DEPTH）。此外，我们提出了一种基于可区分渲染的新型姿势改进方法。主要概念是在多个视图中比较预测并渲染对应关系，以获得与所有视图中预测的对应关系一致的姿势。我们提出的方法对受控设置中的不同数据方式和培训数据类型进行了严格的评估。主要结论是，RGB在对应性估计中表现出色，而如果有良好的3D-3D对应关系，则深度有助于姿势精度。自然，他们的组合可以实现总体最佳性能。我们进行广泛的评估和消融研究，以分析和验证几个具有挑战性的数据集的结果。 DPODV2在所有这些方面都取得了出色的成果，同时仍然保持快速和可扩展性，独立于使用的数据模式和培训数据的类型

本文介绍了一种新型的多视图6 DOF对象姿势细化方法，重点是改进对合成数据训练的方法。它基于DPOD检测器，该检测器会在每个帧中产生密集的2D-3D对应关系。我们选择使用多个具有已知相机转换的帧，因为它允许通过可解释的ICP样损耗函数引入几何约束。损耗函数是通过可区分的渲染器实现的，并经过迭代进行了优化。我们还证明，仅根据合成数据训练的完整检测和完善管道可用于自动标记的真实数据。我们对linemod，caslusion，自制和YCB-V数据集执行定量评估，并与对合成和真实数据训练的最新方法相比，报告出色的性能。我们从经验上证明，我们的方法仅需要几个帧，并且可以在外部摄像机校准中关闭相机位置和噪音，从而使其实际用法更加容易且无处不在。

综合照片 - 现实图像和视频是计算机图形的核心，并且是几十年的研究焦点。传统上，使用渲染算法（如光栅化或射线跟踪）生成场景的合成图像，其将几何形状和材料属性的表示为输入。统称，这些输入定义了实际场景和呈现的内容，并且被称为场景表示（其中场景由一个或多个对象组成）。示例场景表示是具有附带纹理的三角形网格（例如，由艺术家创建），点云（例如，来自深度传感器），体积网格（例如，来自CT扫描）或隐式曲面函数（例如，截短的符号距离）字段）。使用可分辨率渲染损耗的观察结果的这种场景表示的重建被称为逆图形或反向渲染。神经渲染密切相关，并将思想与经典计算机图形和机器学习中的思想相结合，以创建用于合成来自真实观察图像的图像的算法。神经渲染是朝向合成照片现实图像和视频内容的目标的跨越。近年来，我们通过数百个出版物显示了这一领域的巨大进展，这些出版物显示了将被动组件注入渲染管道的不同方式。这种最先进的神经渲染进步的报告侧重于将经典渲染原则与学习的3D场景表示结合的方法，通常现在被称为神经场景表示。这些方法的一个关键优势在于它们是通过设计的3D-一致，使诸如新颖的视点合成捕获场景的应用。除了处理静态场景的方法外，我们还涵盖了用于建模非刚性变形对象的神经场景表示...

We introduce an approach for recovering the 6D pose of multiple known objects in a scene captured by a set of input images with unknown camera viewpoints. First, we present a single-view single-object 6D pose estimation method, which we use to generate 6D object pose hypotheses. Second, we develop a robust method for matching individual 6D object pose hypotheses across different input images in order to jointly estimate camera viewpoints and 6D poses of all objects in a single consistent scene. Our approach explicitly handles object symmetries, does not require depth measurements, is robust to missing or incorrect object hypotheses, and automatically recovers the number of objects in the scene. Third, we develop a method for global scene refinement given multiple object hypotheses and their correspondences across views. This is achieved by solving an object-level bundle adjustment problem that refines the poses of cameras and objects to minimize the reprojection error in all views. We demonstrate that the proposed method, dubbed Cosy-Pose, outperforms current state-of-the-art results for single-view and multi-view 6D object pose estimation by a large margin on two challenging benchmarks: the YCB-Video and T-LESS datasets. Code and pre-trained models are available on the project webpage. 5

传统上，本征成像或内在图像分解被描述为将图像分解为两层：反射率，材料的反射率;和一个阴影，由光和几何之间的相互作用产生。近年来，深入学习技术已广泛应用，以提高这些分离的准确性。在本调查中，我们概述了那些在知名内在图像数据集和文献中使用的相关度量的结果，讨论了预测所需的内在图像分解的适用性。虽然Lambertian的假设仍然是许多方法的基础，但我们表明，对图像形成过程更复杂的物理原理组件的潜力越来越意识到，这是光学准确的材料模型和几何形状，更完整的逆轻型运输估计。考虑使用的前瞻和模型以及驾驶分解过程的学习架构和方法，我们将这些方法分类为分解的类型。考虑到最近神经，逆和可微分的渲染技术的进步，我们还提供了关于未来研究方向的见解。

人类性能捕获是一种非常重要的计算机视觉问题，在电影制作和虚拟/增强现实中具有许多应用。许多以前的性能捕获方法需要昂贵的多视图设置，或者没有恢复具有帧到帧对应关系的密集时空相干几何。我们提出了一种新颖的深度致密人体性能捕获的深层学习方法。我们的方法是基于多视图监督的弱监督方式培训，完全删除了使用3D地面真理注释的培训数据的需求。网络架构基于两个单独的网络，将任务解散为姿势估计和非刚性表面变形步骤。广泛的定性和定量评估表明，我们的方法在质量和稳健性方面优于现有技术。这项工作是DeepCAP的扩展版本，在那里我们提供更详细的解释，比较和结果以及应用程序。

由于在扫描和量化过程中引入了不可避免的噪声，因此通过RGB-D传感器进行的3D重建在几何和纹理中遇到了错误，导致了诸如摄像机漂移，网格失真，纹理幽灵和模糊之类的伪像。考虑到不完美的重建3D模型，大多数以前的方法都集中在几何，纹理或摄像头姿势的完善上。或在以前的关节优化方法中使用了不同的优化方案和优化每个组件的目标，形成了复杂的系统。在本文中，我们提出了一种基于可区分渲染的新型优化方法，该方法通过在渲染结果与相应的RGB-D输入之间执行一致性，将相机姿势，几何形状和纹理的优化整合到统一框架中。基于统一的框架，我们引入了一种联合优化方法，以完全利用几何，纹理和摄像头之间的相互关系，并描述一种自适应交织策略，以提高优化稳定性和效率。使用可区分的渲染，应用图像级的对抗损失用于进一步改善3D模型，从而使其更加逼真。使用定量和定性评估进行合成和真实数据的实验证明了我们在恢复高尺度几何形状和高保真质地方面的优越性。

In this paper, we introduce neural texture learning for 6D object pose estimation from synthetic data and a few unlabelled real images. Our major contribution is a novel learning scheme which removes the drawbacks of previous works, namely the strong dependency on co-modalities or additional refinement. These have been previously necessary to provide training signals for convergence. We formulate such a scheme as two sub-optimisation problems on texture learning and pose learning. We separately learn to predict realistic texture of objects from real image collections and learn pose estimation from pixel-perfect synthetic data. Combining these two capabilities allows then to synthesise photorealistic novel views to supervise the pose estimator with accurate geometry. To alleviate pose noise and segmentation imperfection present during the texture learning phase, we propose a surfel-based adversarial training loss together with texture regularisation from synthetic data. We demonstrate that the proposed approach significantly outperforms the recent state-of-the-art methods without ground-truth pose annotations and demonstrates substantial generalisation improvements towards unseen scenes. Remarkably, our scheme improves the adopted pose estimators substantially even when initialised with much inferior performance.

We introduce MegaPose, a method to estimate the 6D pose of novel objects, that is, objects unseen during training. At inference time, the method only assumes knowledge of (i) a region of interest displaying the object in the image and (ii) a CAD model of the observed object. The contributions of this work are threefold. First, we present a 6D pose refiner based on a render&compare strategy which can be applied to novel objects. The shape and coordinate system of the novel object are provided as inputs to the network by rendering multiple synthetic views of the object's CAD model. Second, we introduce a novel approach for coarse pose estimation which leverages a network trained to classify whether the pose error between a synthetic rendering and an observed image of the same object can be corrected by the refiner. Third, we introduce a large-scale synthetic dataset of photorealistic images of thousands of objects with diverse visual and shape properties and show that this diversity is crucial to obtain good generalization performance on novel objects. We train our approach on this large synthetic dataset and apply it without retraining to hundreds of novel objects in real images from several pose estimation benchmarks. Our approach achieves state-of-the-art performance on the ModelNet and YCB-Video datasets. An extensive evaluation on the 7 core datasets of the BOP challenge demonstrates that our approach achieves performance competitive with existing approaches that require access to the target objects during training. Code, dataset and trained models are available on the project page: https://megapose6d.github.io/.

在本文中，我们解决了多视图3D形状重建的问题。尽管最近与隐式形状表示相关的最新可区分渲染方法提供了突破性的表现，但它们仍然在计算上很重，并且在估计的几何形状上通常缺乏精确性。为了克服这些局限性，我们研究了一种基于体积的新型表示形式建立的新计算方法，就像在最近的可区分渲染方法中一样，但是用深度图进行了参数化，以更好地实现形状表面。与此表示相关的形状能量可以评估给定颜色图像的3D几何形状，并且不需要外观预测，但在优化时仍然受益于体积整合。在实践中，我们提出了一个隐式形状表示，SRDF基于签名距离，我们通过沿摄像头射线进行参数化。相关的形状能量考虑了深度预测一致性和光度一致性之间的一致性，这是在体积表示内的3D位置。可以考虑各种照片一致先验的基础基线，或者像学习功能一样详细的标准。该方法保留具有深度图的像素准确性，并且可行。我们对标准数据集进行的实验表明，它提供了有关具有隐式形状表示的最新方法以及传统的多视角立体方法的最新结果。

Rendering bridges the gap between 2D vision and 3D scenes by simulating the physical process of image formation. By inverting such renderer, one can think of a learning approach to infer 3D information from 2D images. However, standard graphics renderers involve a fundamental discretization step called rasterization, which prevents the rendering process to be differentiable, hence able to be learned. Unlike the state-of-the-art differentiable renderers [29,19], which only approximate the rendering gradient in the back propagation, we propose a truly differentiable rendering framework that is able to (1) directly render colorized mesh using differentiable functions and (2) back-propagate efficient supervision signals to mesh vertices and their attributes from various forms of image representations, including silhouette, shading and color images. The key to our framework is a novel formulation that views rendering as an aggregation function that fuses the probabilistic contributions of all mesh triangles with respect to the rendered pixels. Such formulation enables our framework to flow gradients to the occluded and far-range vertices, which cannot be achieved by the previous state-of-thearts. We show that by using the proposed renderer, one can achieve significant improvement in 3D unsupervised singleview reconstruction both qualitatively and quantitatively. Experiments also demonstrate that our approach is able to handle the challenging tasks in image-based shape fitting, which remain nontrivial to existing differentiable renderers. Code is available at https://github.com/ ShichenLiu/SoftRas.

We introduce a new dataset, Human3.6M, of 3.6 Million accurate 3D Human poses, acquired by recording the performance of 5 female and 6 male subjects, under 4 different viewpoints, for training realistic human sensing systems and for evaluating the next generation of human pose estimation models and algorithms. Besides increasing the size of the datasets in the current state of the art by several orders of magnitude, we also aim to complement such datasets with a diverse set of motions and poses encountered as part of typical human activities (taking photos, talking on the phone, posing, greeting, eating, etc.), with additional synchronized image, human motion capture and time of flight (depth) data, and with accurate 3D body scans of all the subject actors involved. We also provide controlled mixed reality evaluation scenarios where 3D human models are animated using motion capture and inserted using correct 3D geometry, in complex real environments, viewed with moving cameras, and under occlusion. Finally, we provide a set of large scale statistical models and detailed evaluation baselines for the dataset illustrating its diversity and the scope for improvement by future work in the research community. Our experiments show that our best large scale model can leverage our full training set to obtain a 20% improvement in performance compared to a training set of the scale of the largest existing public dataset for this problem. Yet the potential for improvement by leveraging higher capacity, more complex models with our large dataset, is substantially vaster and should stimulate future research. The dataset together with code for the associated large-scale learning models, features, visualization tools, as well as the evaluation server, is available online at http://vision.imar.ro/human3.6m.

Indoor scenes typically exhibit complex, spatially-varying appearance from global illumination, making inverse rendering a challenging ill-posed problem. This work presents an end-to-end, learning-based inverse rendering framework incorporating differentiable Monte Carlo raytracing with importance sampling. The framework takes a single image as input to jointly recover the underlying geometry, spatially-varying lighting, and photorealistic materials. Specifically, we introduce a physically-based differentiable rendering layer with screen-space ray tracing, resulting in more realistic specular reflections that match the input photo. In addition, we create a large-scale, photorealistic indoor scene dataset with significantly richer details like complex furniture and dedicated decorations. Further, we design a novel out-of-view lighting network with uncertainty-aware refinement leveraging hypernetwork-based neural radiance fields to predict lighting outside the view of the input photo. Through extensive evaluations on common benchmark datasets, we demonstrate superior inverse rendering quality of our method compared to state-of-the-art baselines, enabling various applications such as complex object insertion and material editing with high fidelity. Code and data will be made available at \url{https://jingsenzhu.github.io/invrend}.

用于运动中的人类的新型视图综合是一个具有挑战性的计算机视觉问题，使得诸如自由视视频之类的应用。现有方法通常使用具有多个输入视图，3D监控或预训练模型的复杂设置，这些模型不会概括为新标识。旨在解决这些限制，我们提出了一种新颖的视图综合框架，以从单视图传感器捕获的任何人的看法生成现实渲染，其具有稀疏的RGB-D，类似于低成本深度摄像头，而没有参与者特定的楷模。我们提出了一种架构来学习由基于球体的神经渲染获得的小说视图中的密集功能，并使用全局上下文修复模型创建完整的渲染。此外，增强剂网络利用了整体保真度，即使在原始视图中的遮挡区域中也能够产生细节的清晰渲染。我们展示了我们的方法为单个稀疏RGB-D输入产生高质量的合成和真实人体演员的新颖视图。它概括了看不见的身份，新的姿势，忠实地重建面部表情。我们的方法优于现有人体观测合成方法，并且对不同水平的输入稀疏性具有稳健性。

我们提出了可区分的立体声，这是一种多视图立体方法，可从几乎没有输入视图和嘈杂摄像机中重建形状和纹理。我们将传统的立体定向和现代可区分渲染配对，以构建端到端模型，该模型可以预测具有不同拓扑和形状的物体的纹理3D网眼。我们将立体定向作为优化问题，并通过简单的梯度下降同时更新形状和相机。我们进行了广泛的定量分析，并与传统的多视图立体声技术和基于最先进的学习方法进行比较。我们展示了令人信服的重建，这些重建是在挑战现实世界的场景上，以及具有复杂形状，拓扑和纹理的大量对象类型。项目网页：https：//shubham-goel.github.io/ds/

尽管最近的进步，但是，尽管最近的进展，但是从单个图像中的人类姿势的全3D估计仍然是一个具有挑战性的任务。在本文中，我们探讨了关于场景几何体的强先前信息的假设可用于提高姿态估计精度。为了主弱地解决这个问题，我们已经组装了一种新的$ \ textbf {几何姿势提供} $ DataSet，包括与各种丰富的3D环境交互的人员的多视图图像。我们利用商业运动捕获系统来收集场景本身的姿势和构造精确的几何3D CAD模型的金标估计。要将对现有框架的现有框架注入图像的现有框架，我们介绍了一种新颖的，基于视图的场景几何形状，一个$ \ textbf {多层深度图} $，它采用了多次射线跟踪到简明地编码沿着每种相机视图光线方向的多个表面入口和退出点。我们提出了两种不同的机制，用于集成多层深度信息姿势估计：输入作为升降2D姿势的编码光线特征，其次是促进学习模型以支持几何一致姿态估计的可差异损失。我们通过实验展示这些技术可以提高3D姿势估计的准确性，特别是在遮挡和复杂场景几何形状的存在中。

在本文中，我们考虑了同时找到和从单个2D图像中恢复多手的具有挑战性的任务。先前的研究要么关注单手重建，要么以多阶段的方式解决此问题。此外，常规的两阶段管道首先检测到手部区域，然后估计每个裁剪贴片的3D手姿势。为了减少预处理和特征提取中的计算冗余，我们提出了一条简洁但有效的单阶段管道。具体而言，我们为多手重建设计了多头自动编码器结构，每个HEAD网络分别共享相同的功能图并分别输出手动中心，姿势和纹理。此外，我们采用了一个弱监督的计划来减轻昂贵的3D现实世界数据注释的负担。为此，我们提出了一系列通过舞台训练方案优化的损失，其中根据公开可用的单手数据集生成具有2D注释的多手数据集。为了进一步提高弱监督模型的准确性，我们在单手和多个手设置中采用了几个功能一致性约束。具体而言，从本地功能估算的每只手的关键点应与全局功能预测的重新投影点一致。在包括Freihand，HO3D，Interhand 2.6M和RHD在内的公共基准测试的广泛实验表明，我们的方法在弱监督和完全监督的举止中优于基于最先进的模型方法。代码和模型可在{\ url {https://github.com/zijinxuxu/smhr}}上获得。