卷积神经网络（CNN）已被广泛用于各种视觉任务，例如图像分类，语义分割等。不幸的是，标准2D CNN不太适合球形信号，例如全景图像或球形投影，因为球体是一个非结构化的网格。在本文中，我们提出了球形变压器，可以将球形信号转换为可以通过标准CNN直接处理的向量，从而通过预处理可以在任务和数据集中重复使用许多精心设计的CNNS体系结构。为此，提出的方法首先使用局部结构化采样方法（例如HealPix）通过使用球形点及其相邻点的信息来构建变压器网格，然后通过网格将球形信号转换为向量。通过构建球形变压器模块，我们可以直接使用多个CNN体系结构。我们评估了有关球形MNIST识别，3D对象分类和全向图像语义分割的任务的方法。对于3D对象分类，我们进一步提出了一种基于渲染的投影方法，以提高性能和旋转等值模型，以提高抗旋转能力。关于三个任务的实验结果表明，我们的方法比最先进的方法实现了卓越的性能。

3D shape models are becoming widely available and easier to capture, making available 3D information crucial for progress in object classification. Current state-of-theart methods rely on CNNs to address this problem. Recently, we witness two types of CNNs being developed: CNNs based upon volumetric representations versus CNNs based upon multi-view representations. Empirical results from these two types of CNNs exhibit a large gap, indicating that existing volumetric CNN architectures and approaches are unable to fully exploit the power of 3D representations. In this paper, we aim to improve both volumetric CNNs and multi-view CNNs according to extensive analysis of existing approaches. To this end, we introduce two distinct network architectures of volumetric CNNs. In addition, we examine multi-view CNNs, where we introduce multiresolution filtering in 3D. Overall, we are able to outperform current state-of-the-art methods for both volumetric CNNs and multi-view CNNs. We provide extensive experiments designed to evaluate underlying design choices, thus providing a better understanding of the space of methods available for object classification on 3D data.

点云分析没有姿势前导者在真实应用中非常具有挑战性，因为点云的方向往往是未知的。在本文中，我们提出了一个全新的点集学习框架prin，即点亮旋转不变网络，专注于点云分析中的旋转不变特征提取。我们通过密度意识的自适应采样构建球形信号，以处理球形空间中的扭曲点分布。提出了球形Voxel卷积和点重新采样以提取每个点的旋转不变特征。此外，我们将Prin扩展到称为Sprin的稀疏版本，直接在稀疏点云上运行。 Prin和Sprin都可以应用于从对象分类，部分分割到3D特征匹配和标签对齐的任务。结果表明，在随机旋转点云的数据集上，Sprin比无任何数据增强的最先进方法表现出更好的性能。我们还为我们的方法提供了彻底的理论证明和分析，以实现我们的方法实现的点明智的旋转不变性。我们的代码可在https://github.com/qq456cvb/sprin上找到。

学习3D点云的新表示形式是3D视觉中的一个活跃研究领域，因为订单不变的点云结构仍然对神经网络体系结构的设计构成挑战。最近的作品探索了学习全球或本地功能或两者兼而有之，但是均未通过分析点的局部方向分布来捕获上下文形状信息的早期方法。在本文中，我们利用点附近的点方向分布，以获取点云的表现力局部邻里表示。我们通过将给定点的球形邻域分为预定义的锥体来实现这一目标，并将每个体积内部的统计数据用作点特征。这样，本地贴片不仅可以由所选点的最近邻居表示，还可以考虑沿该点周围多个方向定义的点密度分布。然后，我们能够构建涉及依赖MLP（多层感知器）层的Odfblock的方向分布函数（ODF）神经网络。新的ODFNET模型可实现ModelNet40和ScanObjectNN数据集的对象分类的最新精度，并在Shapenet S3DIS数据集上进行分割。

Point cloud is an important type of geometric data structure. Due to its irregular format, most researchers transform such data to regular 3D voxel grids or collections of images. This, however, renders data unnecessarily voluminous and causes issues. In this paper, we design a novel type of neural network that directly consumes point clouds, which well respects the permutation invariance of points in the input. Our network, named PointNet, provides a unified architecture for applications ranging from object classification, part segmentation, to scene semantic parsing. Though simple, PointNet is highly efficient and effective. Empirically, it shows strong performance on par or even better than state of the art. Theoretically, we provide analysis towards understanding of what the network has learnt and why the network is robust with respect to input perturbation and corruption.

3D点云的卷积经过广泛研究，但在几何深度学习中却远非完美。卷积的传统智慧在3D点之间表现出特征对应关系，这是对差的独特特征学习的内在限制。在本文中，我们提出了自适应图卷积（AGCONV），以供点云分析的广泛应用。 AGCONV根据其动态学习的功能生成自适应核。与使用固定/各向同性核的解决方案相比，AGCONV提高了点云卷积的灵活性，有效，精确地捕获了不同语义部位的点之间的不同关系。与流行的注意力体重方案不同，AGCONV实现了卷积操作内部的适应性，而不是简单地将不同的权重分配给相邻点。广泛的评估清楚地表明，我们的方法优于各种基准数据集中的点云分类和分割的最新方法。同时，AGCONV可以灵活地采用更多的点云分析方法来提高其性能。为了验证其灵活性和有效性，我们探索了基于AGCONV的完成，DeNoing，Upsmpling，注册和圆圈提取的范式，它们与竞争对手相当甚至优越。我们的代码可在https://github.com/hrzhou2/adaptconv-master上找到。

Unlike images which are represented in regular dense grids, 3D point clouds are irregular and unordered, hence applying convolution on them can be difficult. In this paper, we extend the dynamic filter to a new convolution operation, named PointConv. PointConv can be applied on point clouds to build deep convolutional networks. We treat convolution kernels as nonlinear functions of the local coordinates of 3D points comprised of weight and density functions. With respect to a given point, the weight functions are learned with multi-layer perceptron networks and density functions through kernel density estimation. The most important contribution of this work is a novel reformulation proposed for efficiently computing the weight functions, which allowed us to dramatically scale up the network and significantly improve its performance. The learned convolution kernel can be used to compute translation-invariant and permutation-invariant convolution on any point set in the 3D space. Besides, PointConv can also be used as deconvolution operators to propagate features from a subsampled point cloud back to its original resolution. Experiments on ModelNet40, ShapeNet, and ScanNet show that deep convolutional neural networks built on PointConv are able to achieve state-of-the-art on challenging semantic segmentation benchmarks on 3D point clouds. Besides, our experiments converting CIFAR-10 into a point cloud showed that networks built on PointConv can match the performance of convolutional networks in 2D images of a similar structure.

We present a network architecture for processing point clouds that directly operates on a collection of points represented as a sparse set of samples in a high-dimensional lattice. Naïvely applying convolutions on this lattice scales poorly, both in terms of memory and computational cost, as the size of the lattice increases. Instead, our network uses sparse bilateral convolutional layers as building blocks. These layers maintain efficiency by using indexing structures to apply convolutions only on occupied parts of the lattice, and allow flexible specifications of the lattice structure enabling hierarchical and spatially-aware feature learning, as well as joint 2D-3D reasoning. Both point-based and image-based representations can be easily incorporated in a network with such layers and the resulting model can be trained in an end-to-end manner. We present results on 3D segmentation tasks where our approach outperforms existing state-of-the-art techniques.

Point cloud learning has lately attracted increasing attention due to its wide applications in many areas, such as computer vision, autonomous driving, and robotics. As a dominating technique in AI, deep learning has been successfully used to solve various 2D vision problems. However, deep learning on point clouds is still in its infancy due to the unique challenges faced by the processing of point clouds with deep neural networks. Recently, deep learning on point clouds has become even thriving, with numerous methods being proposed to address different problems in this area. To stimulate future research, this paper presents a comprehensive review of recent progress in deep learning methods for point clouds. It covers three major tasks, including 3D shape classification, 3D object detection and tracking, and 3D point cloud segmentation. It also presents comparative results on several publicly available datasets, together with insightful observations and inspiring future research directions.

随着激光雷达传感器和3D视觉摄像头的扩散，3D点云分析近年来引起了重大关注。经过先驱工作点的成功后，基于深度学习的方法越来越多地应用于各种任务，包括3D点云分段和3D对象分类。在本文中，我们提出了一种新颖的3D点云学习网络，通过选择性地执行具有动态池的邻域特征聚合和注意机制来提出作为动态点特征聚合网络（DPFA-NET）。 DPFA-Net有两个可用于三维云的语义分割和分类的变体。作为DPFA-NET的核心模块，我们提出了一个特征聚合层，其中每个点的动态邻域的特征通过自我注意机制聚合。与其他分割模型相比，来自固定邻域的聚合特征，我们的方法可以在不同层中聚合来自不同邻居的特征，在不同层中为查询点提供更具选择性和更广泛的视图，并更多地关注本地邻域中的相关特征。此外，为了进一步提高所提出的语义分割模型的性能，我们提出了两种新方法，即两级BF-Net和BF-Rengralization来利用背景前台信息。实验结果表明，所提出的DPFA-Net在S3DIS数据集上实现了最先进的整体精度分数，在S3DIS数据集上进行了语义分割，并在不同的语义分割，部分分割和3D对象分类中提供始终如一的令人满意的性能。与其他方法相比，它也在计算上更有效。

我们介绍了PointConvormer，这是一个基于点云的深神经网络体系结构的新颖构建块。受到概括理论的启发，PointConvormer结合了点卷积的思想，其中滤波器权重仅基于相对位置，而变形金刚则利用了基于功能的注意力。在PointConvormer中，附近点之间的特征差异是重量重量卷积权重的指标。因此，我们从点卷积操作中保留了不变，而注意力被用来选择附近的相关点进行卷积。为了验证PointConvormer的有效性，我们在点云上进行了语义分割和场景流估计任务，其中包括扫描仪，Semantickitti，FlyingThings3D和Kitti。我们的结果表明，PointConvormer具有经典的卷积，常规变压器和Voxelized稀疏卷积方法的表现，具有较小，更高效的网络。可视化表明，PointConvormer的性能类似于在平面表面上的卷积，而邻域选择效果在物体边界上更强，表明它具有两全其美。

多视图投影方法在3D理解任务等方面表现出有希望的性能，如3D分类和分割。然而，它仍然不明确如何将这种多视图方法与广泛可用的3D点云组合。以前的方法使用未受忘掉的启发式方法在点级别结合功能。为此，我们介绍了多视图点云（vinoint云）的概念，表示每个3D点作为从多个视图点提取的一组功能。这种新颖的3D Vintor云表示将3D点云表示的紧凑性与多视图表示的自然观。当然，我们可以用卷积和汇集操作配备这一新的表示。我们以理论上建立的功能形式部署了Voint神经网络（vointnet），以学习vinite空间中的表示。我们的小说代表在ScanObjectnn，ModelNet40和ShapEnet​​ Core55上实现了3D分类和检索的最先进的性能。此外，我们在ShapeNet零件上实现了3D语义细分的竞争性能。进一步的分析表明，与其他方法相比，求力提高了旋转和闭塞的鲁棒性。

点云识别是工业机器人和自主驾驶中的重要任务。最近，几个点云处理模型已经实现了最先进的表演。然而，这些方法缺乏旋转稳健性，并且它们的性能严重降低了随机旋转，未能扩展到具有不同方向的现实情景。为此，我们提出了一种名为基于自行轮廓的转换（SCT）的方法，该方法可以灵活地集成到针对任意旋转的各种现有点云识别模型中。 SCT通过引入轮廓感知的转换（CAT）提供有效的旋转和翻译不变性，该转换（CAT）线性地将点数的笛卡尔坐标转换为翻译和旋转 - 不变表示。我们证明猫是一种基于理论分析的旋转和翻译不变的转换。此外，提出了帧对准模块来增强通过捕获轮廓并将基于自平台的帧转换为帧内帧来增强鉴别特征提取。广泛的实验结果表明，SCT在合成和现实世界基准的有效性和效率的任意旋转下表现出最先进的方法。此外，稳健性和一般性评估表明SCT是稳健的，适用于各种点云处理模型，它突出了工业应用中SCT的优势。

全景图像可以同时展示周围环境的完整信息，并且在虚拟旅游，游戏，机器人技术等方面具有许多优势。但是，全景深度估计的进度无法完全解决由常用的投射方法引起的失真和不连续性问题。本文提出了SphereDepth，这是一种新型的全景深度估计方法，该方法可直接预测球形网格的深度而无需投影预处理。核心思想是建立全景图像与球形网格之间的关系，然后使用深层神经网络在球形域上提取特征以预测深度。为了解决高分辨率全景数据带来的效率挑战，我们介绍了两个超参数，以平衡推理速度和准确性。在三个公共全景数据集中验证，SphereDepth通过全景深度估算的最新方法实现了可比的结果。从球形域设置中受益，球形部可以产生高质量的点云，并显着缓解失真和不连续性问题。

基于2D图像的3D对象的推理由于从不同方向查看对象引起的外观差异很大，因此具有挑战性。理想情况下，我们的模型将是对物体姿势变化的不变或等效的。不幸的是，对于2D图像输入，这通常是不可能的，因为我们没有一个先验模型，即在平面外对象旋转下如何改变图像。唯一的$ \ mathrm {so}（3）$ - 当前存在的模型需要点云输入而不是2D图像。在本文中，我们提出了一种基于Icosahedral群卷积的新型模型体系结构，即通过将输入图像投影到iCosahedron上，以$ \ mathrm {so（3）} $中的理由。由于此投影，该模型大致与$ \ mathrm {so}（3）$中的旋转大致相当。我们将此模型应用于对象构成估计任务，并发现它的表现优于合理的基准。

Point clouds are characterized by irregularity and unstructuredness, which pose challenges in efficient data exploitation and discriminative feature extraction. In this paper, we present an unsupervised deep neural architecture called Flattening-Net to represent irregular 3D point clouds of arbitrary geometry and topology as a completely regular 2D point geometry image (PGI) structure, in which coordinates of spatial points are captured in colors of image pixels. \mr{Intuitively, Flattening-Net implicitly approximates a locally smooth 3D-to-2D surface flattening process while effectively preserving neighborhood consistency.} \mr{As a generic representation modality, PGI inherently encodes the intrinsic property of the underlying manifold structure and facilitates surface-style point feature aggregation.} To demonstrate its potential, we construct a unified learning framework directly operating on PGIs to achieve \mr{diverse types of high-level and low-level} downstream applications driven by specific task networks, including classification, segmentation, reconstruction, and upsampling. Extensive experiments demonstrate that our methods perform favorably against the current state-of-the-art competitors. We will make the code and data publicly available at https://github.com/keeganhk/Flattening-Net.

We present an approach to semantic scene analysis using deep convolutional networks. Our approach is based on tangent convolutions -a new construction for convolutional networks on 3D data. In contrast to volumetric approaches, our method operates directly on surface geometry. Crucially, the construction is applicable to unstructured point clouds and other noisy real-world data. We show that tangent convolutions can be evaluated efficiently on large-scale point clouds with millions of points. Using tangent convolutions, we design a deep fully-convolutional network for semantic segmentation of 3D point clouds, and apply it to challenging real-world datasets of indoor and outdoor 3D environments. Experimental results show that the presented approach outperforms other recent deep network constructions in detailed analysis of large 3D scenes.

Multi-view projection techniques have shown themselves to be highly effective in achieving top-performing results in the recognition of 3D shapes. These methods involve learning how to combine information from multiple view-points. However, the camera view-points from which these views are obtained are often fixed for all shapes. To overcome the static nature of current multi-view techniques, we propose learning these view-points. Specifically, we introduce the Multi-View Transformation Network (MVTN), which uses differentiable rendering to determine optimal view-points for 3D shape recognition. As a result, MVTN can be trained end-to-end with any multi-view network for 3D shape classification. We integrate MVTN into a novel adaptive multi-view pipeline that is capable of rendering both 3D meshes and point clouds. Our approach demonstrates state-of-the-art performance in 3D classification and shape retrieval on several benchmarks (ModelNet40, ScanObjectNN, ShapeNet Core55). Further analysis indicates that our approach exhibits improved robustness to occlusion compared to other methods. We also investigate additional aspects of MVTN, such as 2D pretraining and its use for segmentation. To support further research in this area, we have released MVTorch, a PyTorch library for 3D understanding and generation using multi-view projections.

注意机制在点云分析中发挥了越来越重要的作用，并且渠道注意是热点之一。通过这么多的频道信息，神经网络难以筛选有用的信道信息。因此，提出了一种自适应信道编码机制以在本文中捕获信道关系。它通过明确地编码其特征信道之间的相互依赖来提高网络生成的表示的质量。具体地，提出了一种通道 - 明智的卷积（通道-Chim）以自适应地学习坐标和特征之间的关系，以便编码信道。与流行的重量方案不同，本文提出的通道CONN实现了卷积操作的适应性，而不是简单地为频道分配不同的权重。对现有基准的广泛实验验证了我们的方法实现了艺术的状态。

This paper presents SO-Net, a permutation invariant architecture for deep learning with orderless point clouds. The SO-Net models the spatial distribution of point cloud by building a Self-Organizing Map (SOM). Based on the SOM, SO-Net performs hierarchical feature extraction on individual points and SOM nodes, and ultimately represents the input point cloud by a single feature vector. The receptive field of the network can be systematically adjusted by conducting point-to-node k nearest neighbor search. In recognition tasks such as point cloud reconstruction, classification, object part segmentation and shape retrieval, our proposed network demonstrates performance that is similar with or better than state-of-the-art approaches. In addition, the training speed is significantly faster than existing point cloud recognition networks because of the parallelizability and simplicity of the proposed architecture. Our code is