3D object detection with surround-view images is an essential task for autonomous driving. In this work, we propose DETR4D, a Transformer-based framework that explores sparse attention and direct feature query for 3D object detection in multi-view images. We design a novel projective cross-attention mechanism for query-image interaction to address the limitations of existing methods in terms of geometric cue exploitation and information loss for cross-view objects. In addition, we introduce a heatmap generation technique that bridges 3D and 2D spaces efficiently via query initialization. Furthermore, unlike the common practice of fusing intermediate spatial features for temporal aggregation, we provide a new perspective by introducing a novel hybrid approach that performs cross-frame fusion over past object queries and image features, enabling efficient and robust modeling of temporal information. Extensive experiments on the nuScenes dataset demonstrate the effectiveness and efficiency of the proposed DETR4D.
translated by 谷歌翻译
多尺度功能已被证明在对象检测方面非常有效,大多数基于Convnet的对象检测器采用特征金字塔网络(FPN)作为利用多尺度功能的基本组件。但是,对于最近提出的基于变压器的对象探测器,直接结合多尺度功能会导致由于处理高分辨率特征的注意机制的高复杂性,因此导致了高度的计算开销。本文介绍了迭代多尺度特征聚合(IMFA) - 一种通用范式,可有效利用基于变压器的对象检测器中的多尺度特征。核心想法是从仅几个关键位置利用稀疏的多尺度特征,并且通过两种新颖的设计实现了稀疏的特征。首先,IMFA重新安排变压器编码器数据管道,因此可以根据检测预测进行迭代更新编码的功能。其次,在先前检测预测的指导下,IMFA稀疏的量表自适应特征可从几个关键点位置进行精制检测。结果,采样的多尺度特征稀疏,但仍然对对象检测非常有益。广泛的实验表明,提出的IMFA在略有计算开销的情况下显着提高了基于变压器的对象检测器的性能。项目页面:https://github.com/zhanggongjie/imfa。
translated by 谷歌翻译
随着LIDAR传感器在自动驾驶中的流行率,3D对象跟踪受到了越来越多的关注。在点云序列中,3D对象跟踪旨在预测给定对象模板中连续帧中对象的位置和方向。在变压器成功的驱动下,我们提出了点跟踪变压器(PTTR),它有效地预测了高质量的3D跟踪,借助变压器操作,以粗到1的方式导致。 PTTR由三个新型设计组成。 1)我们设计的关系意识采样代替随机抽样,以在亚采样过程中保留与给定模板相关的点。 2)我们提出了一个点关系变压器,以进行有效的特征聚合和模板和搜索区域之间的特征匹配。 3)基于粗糙跟踪结果,我们采用了一个新颖的预测改进模块,通过局部特征池获得最终的完善预测。此外,以捕获对象运动的鸟眼视图(BEV)的有利特性(BEV)的良好属性,我们进一步设计了一个名为PTTR ++的更高级的框架,该框架既包含了点的视图和BEV表示)产生高质量跟踪结果的影响。 PTTR ++实质上提高了PTTR顶部的跟踪性能,并具有低计算开销。多个数据集的广泛实验表明,我们提出的方法达到了卓越的3D跟踪准确性和效率。
translated by 谷歌翻译
使用点云的3D对象检测由于其在自动驾驶和机器人技术中的广泛应用而引起了越来越多的关注。但是,大多数现有的研究都集中在单点云框架上,而无需利用点云序列中的时间信息。在本文中,我们设计了Transpillars,这是一种基于变压器的新型特征聚合技术,可利用连续点云框架的时间特征用于多帧3D对象检测。从两个角度来看,转子汇总的时空点云特征。首先,它直接从多帧特征映射而不是汇总实例功能融合体素级特征,以保存实例详细信息,并使用上下文信息,这些信息对于准确的对象本地化至关重要。其次,它引入了分层的粗到精细策略,以逐步融合多尺度功能,以有效捕获移动对象的运动并指导精美特征的聚合。此外,引入了一系列可变形变压器,以提高跨帧功能匹配的有效性。广泛的实验表明,与现有的多帧检测方法相比,我们提议的转质质量可以达到最先进的性能。代码将发布。
translated by 谷歌翻译
通过将元学习纳入基于区域的检测框架中,很少有射击对象检测经过广泛的研究。尽管取得了成功,但所述范式仍然受到几个因素的限制,例如(i)新型类别的低质量区域建议以及(ii)不同类别之间的类间相关性的过失。这种限制阻碍了基础知识的概括,以检测新型级别对象。在这项工作中,我们设计了元数据,(i)是第一个图像级的少量检测器,(ii)引入了一种新颖的类间相关元学习策略,以捕获和利用不同类别之间的相关性的相关性稳健而准确的几个射击对象检测。 meta-detr完全在图像级别工作,没有任何区域建议,这规避了普遍的几杆检测框架中不准确的建议的约束。此外,引入的相关元学习使元数据能够同时参加单个进料中的多个支持类别,从而可以捕获不同类别之间的类间相关性,从而大大降低了相似类别的错误分类并增强知识概括性参加新颖的课程。对多个射击对象检测基准进行的实验表明,所提出的元元删除优于大幅度的最先进方法。实施代码可在https://github.com/zhanggongjie/meta-detr上获得。
translated by 谷歌翻译
最近提出的检测变压器(DETR)已建立了一个完全端到端的范式以进行对象检测。但是,DETR遭受慢训练的融合,这阻碍了其对各种检测任务的适用性。我们观察到,由于对象查询和编码图像特征之间的语义不一致,DETR的缓慢收敛在很大程度上归因于将对象查询与相关区域匹配的困难。通过此观察,我们设计了与DETR ++(SAM-DETR ++)设计的语义对齐匹配,以加速DETR的收敛并改善检测性能。 SAM-DETR ++的核心是一个插件模块,该模块将对象查询和编码图像功能投射到相同的功能嵌入空间中,在该空间中,每个对象查询都可以轻松地与具有相似语义的相关区域匹配。此外,SAM-DETR ++搜索了多个代表性关键点,并利用其功能以具有增强的表示能力的语义对齐匹配。此外,SAM-DETR ++可以根据设计的语义对准匹配,以粗到5的方式有效地融合多尺度特征。广泛的实验表明,所提出的SAM-DETR ++实现了优越的收敛速度和竞争性检测准确性。此外,作为一种插件方法,SAM-DETR ++可以以更好的性能补充现有的DITR收敛解决方案,仅使用12个训练时代获得44.8%的AP和49.1%的AP,并使用Resnet-50上的CoCo Val2017上的50个训练时代获得50个训练时期。代码可在https://github.com/zhanggongjie/sam-detr上找到。
translated by 谷歌翻译
培训有效的生成对抗性网络(GANS)需要大量的培训数据,但是训练型模型通常是用鉴别器过度拟合的次优。通过大规模和手工制作的数据增强,通过扩大有限培训数据的分布来解决此问题的几项问题。我们从一个非常不同的角度处理数据限制图像生成。具体而言,我们设计Genco,这是一种生成的共同培训网络,通过引入多种互补鉴别者来减轻鉴别者过度拟合问题,这些判别符号在培训中提供多种独特的观点来提供不同的监督。我们以两种方式实例化了Genco的想法。首先是重量差异共同训练(WECO),其通过多样化它们的参数共同列举多个独特的鉴别器。第二种方式是数据差异共同训练(DACO),其通过馈送具有输入图像的不同视图的鉴别器(例如,输入图像的不同频率分量)来实现共同训练。在多个基准上进行广泛的实验表明,Genco实现了具有有限培训数据的优异发电。此外,Genco还通过组合时补充了增强方法,并在结合时进行了一致和明确的性能。
translated by 谷歌翻译
Masked image modeling (MIM) performs strongly in pre-training large vision Transformers (ViTs). However, small models that are critical for real-world applications cannot or only marginally benefit from this pre-training approach. In this paper, we explore distillation techniques to transfer the success of large MIM-based pre-trained models to smaller ones. We systematically study different options in the distillation framework, including distilling targets, losses, input, network regularization, sequential distillation, etc, revealing that: 1) Distilling token relations is more effective than CLS token- and feature-based distillation; 2) An intermediate layer of the teacher network as target perform better than that using the last layer when the depth of the student mismatches that of the teacher; 3) Weak regularization is preferred; etc. With these findings, we achieve significant fine-tuning accuracy improvements over the scratch MIM pre-training on ImageNet-1K classification, using all the ViT-Tiny, ViT-Small, and ViT-base models, with +4.2%/+2.4%/+1.4% gains, respectively. Our TinyMIM model of base size achieves 52.2 mIoU in AE20K semantic segmentation, which is +4.1 higher than the MAE baseline. Our TinyMIM model of tiny size achieves 79.6% top-1 accuracy on ImageNet-1K image classification, which sets a new record for small vision models of the same size and computation budget. This strong performance suggests an alternative way for developing small vision Transformer models, that is, by exploring better training methods rather than introducing inductive biases into architectures as in most previous works. Code is available at https://github.com/OliverRensu/TinyMIM.
translated by 谷歌翻译
In this paper, we propose a robust 3D detector, named Cross Modal Transformer (CMT), for end-to-end 3D multi-modal detection. Without explicit view transformation, CMT takes the image and point clouds tokens as inputs and directly outputs accurate 3D bounding boxes. The spatial alignment of multi-modal tokens is performed implicitly, by encoding the 3D points into multi-modal features. The core design of CMT is quite simple while its performance is impressive. CMT obtains 73.0% NDS on nuScenes benchmark. Moreover, CMT has a strong robustness even if the LiDAR is missing. Code will be released at https://github.com/junjie18/CMT.
translated by 谷歌翻译
Dataset distillation has emerged as a prominent technique to improve data efficiency when training machine learning models. It encapsulates the knowledge from a large dataset into a smaller synthetic dataset. A model trained on this smaller distilled dataset can attain comparable performance to a model trained on the original training dataset. However, the existing dataset distillation techniques mainly aim at achieving the best trade-off between resource usage efficiency and model utility. The security risks stemming from them have not been explored. This study performs the first backdoor attack against the models trained on the data distilled by dataset distillation models in the image domain. Concretely, we inject triggers into the synthetic data during the distillation procedure rather than during the model training stage, where all previous attacks are performed. We propose two types of backdoor attacks, namely NAIVEATTACK and DOORPING. NAIVEATTACK simply adds triggers to the raw data at the initial distillation phase, while DOORPING iteratively updates the triggers during the entire distillation procedure. We conduct extensive evaluations on multiple datasets, architectures, and dataset distillation techniques. Empirical evaluation shows that NAIVEATTACK achieves decent attack success rate (ASR) scores in some cases, while DOORPING reaches higher ASR scores (close to 1.0) in all cases. Furthermore, we conduct a comprehensive ablation study to analyze the factors that may affect the attack performance. Finally, we evaluate multiple defense mechanisms against our backdoor attacks and show that our attacks can practically circumvent these defense mechanisms.
translated by 谷歌翻译