Masked image modeling (MIM) performs strongly in pre-training large vision Transformers (ViTs). However, small models that are critical for real-world applications cannot or only marginally benefit from this pre-training approach. In this paper, we explore distillation techniques to transfer the success of large MIM-based pre-trained models to smaller ones. We systematically study different options in the distillation framework, including distilling targets, losses, input, network regularization, sequential distillation, etc, revealing that: 1) Distilling token relations is more effective than CLS token- and feature-based distillation; 2) An intermediate layer of the teacher network as target perform better than that using the last layer when the depth of the student mismatches that of the teacher; 3) Weak regularization is preferred; etc. With these findings, we achieve significant fine-tuning accuracy improvements over the scratch MIM pre-training on ImageNet-1K classification, using all the ViT-Tiny, ViT-Small, and ViT-base models, with +4.2%/+2.4%/+1.4% gains, respectively. Our TinyMIM model of base size achieves 52.2 mIoU in AE20K semantic segmentation, which is +4.1 higher than the MAE baseline. Our TinyMIM model of tiny size achieves 79.6% top-1 accuracy on ImageNet-1K image classification, which sets a new record for small vision models of the same size and computation budget. This strong performance suggests an alternative way for developing small vision Transformer models, that is, by exploring better training methods rather than introducing inductive biases into architectures as in most previous works. Code is available at https://github.com/OliverRensu/TinyMIM.
translated by 谷歌翻译
The number of international benchmarking competitions is steadily increasing in various fields of machine learning (ML) research and practice. So far, however, little is known about the common practice as well as bottlenecks faced by the community in tackling the research questions posed. To shed light on the status quo of algorithm development in the specific field of biomedical imaging analysis, we designed an international survey that was issued to all participants of challenges conducted in conjunction with the IEEE ISBI 2021 and MICCAI 2021 conferences (80 competitions in total). The survey covered participants' expertise and working environments, their chosen strategies, as well as algorithm characteristics. A median of 72% challenge participants took part in the survey. According to our results, knowledge exchange was the primary incentive (70%) for participation, while the reception of prize money played only a minor role (16%). While a median of 80 working hours was spent on method development, a large portion of participants stated that they did not have enough time for method development (32%). 25% perceived the infrastructure to be a bottleneck. Overall, 94% of all solutions were deep learning-based. Of these, 84% were based on standard architectures. 43% of the respondents reported that the data samples (e.g., images) were too large to be processed at once. This was most commonly addressed by patch-based training (69%), downsampling (37%), and solving 3D analysis tasks as a series of 2D tasks. K-fold cross-validation on the training set was performed by only 37% of the participants and only 50% of the participants performed ensembling based on multiple identical models (61%) or heterogeneous models (39%). 48% of the respondents applied postprocessing steps.
translated by 谷歌翻译
Accurate airway extraction from computed tomography (CT) images is a critical step for planning navigation bronchoscopy and quantitative assessment of airway-related chronic obstructive pulmonary disease (COPD). The existing methods are challenging to sufficiently segment the airway, especially the high-generation airway, with the constraint of the limited label and cannot meet the clinical use in COPD. We propose a novel two-stage 3D contextual transformer-based U-Net for airway segmentation using CT images. The method consists of two stages, performing initial and refined airway segmentation. The two-stage model shares the same subnetwork with different airway masks as input. Contextual transformer block is performed both in the encoder and decoder path of the subnetwork to finish high-quality airway segmentation effectively. In the first stage, the total airway mask and CT images are provided to the subnetwork, and the intrapulmonary airway mask and corresponding CT scans to the subnetwork in the second stage. Then the predictions of the two-stage method are merged as the final prediction. Extensive experiments were performed on in-house and multiple public datasets. Quantitative and qualitative analysis demonstrate that our proposed method extracted much more branches and lengths of the tree while accomplishing state-of-the-art airway segmentation performance. The code is available at https://github.com/zhaozsq/airway_segmentation.
translated by 谷歌翻译
Deep learning (DL)-based tomographic SAR imaging algorithms are gradually being studied. Typically, they use an unfolding network to mimic the iterative calculation of the classical compressive sensing (CS)-based methods and process each range-azimuth unit individually. However, only one-dimensional features are effectively utilized in this way. The correlation between adjacent resolution units is ignored directly. To address that, we propose a new model-data-driven network to achieve tomoSAR imaging based on multi-dimensional features. Guided by the deep unfolding methodology, a two-dimensional deep unfolding imaging network is constructed. On the basis of it, we add two 2D processing modules, both convolutional encoder-decoder structures, to enhance multi-dimensional features of the imaging scene effectively. Meanwhile, to train the proposed multifeature-based imaging network, we construct a tomoSAR simulation dataset consisting entirely of simulation data of buildings. Experiments verify the effectiveness of the model. Compared with the conventional CS-based FISTA method and DL-based gamma-Net method, the result of our proposed method has better performance on completeness while having decent imaging accuracy.
translated by 谷歌翻译
Object re-identification method is made up of backbone network, feature aggregation, and loss function. However, most backbone networks lack a special mechanism to handle rich scale variations and mine discriminative feature representations. In this paper, we firstly design a hierarchical similarity graph module (HSGM) to reduce the conflict of backbone and re-identification networks. The designed HSGM builds a rich hierarchical graph to mine the mapping relationships between global-local and local-local. Secondly, we divide the feature map along with the spatial and channel directions in each hierarchical graph. The HSGM applies the spatial features and channel features extracted from different locations as nodes, respectively, and utilizes the similarity scores between nodes to construct spatial and channel similarity graphs. During the learning process of HSGM, we utilize a learnable parameter to re-optimize the importance of each position, as well as evaluate the correlation between different nodes. Thirdly, we develop a novel hierarchical similarity graph network (HSGNet) by embedding the HSGM in the backbone network. Furthermore, HSGM can be easily embedded into backbone networks of any depth to improve object re-identification ability. Finally, extensive experiments on three large-scale object datasets demonstrate that the proposed HSGNet is superior to state-of-the-art object re-identification approaches.
translated by 谷歌翻译
预测公路参与者的未来运动对于自动驾驶至关重要,但由于令人震惊的运动不确定性,因此极具挑战性。最近,大多数运动预测方法求助于基于目标的策略,即预测运动轨迹的终点,作为回归整个轨迹的条件,以便可以减少解决方案的搜索空间。但是,准确的目标坐标很难预测和评估。此外,目的地的点表示限制了丰富的道路环境的利用,从而导致预测不准确。目标区域,即可能的目的地区域,而不是目标坐标,可以通过涉及更多的容忍度和指导来提供更软的限制,以搜索潜在的轨迹。考虑到这一点,我们提出了一个新的基于目标区域的框架,名为“目标区域网络”(GANET)进行运动预测,该框架对目标区域进行了建模,而不是确切的目标坐标作为轨迹预测的先决条件,更加可靠,更准确地执行。具体而言,我们建议一个goicrop(目标的目标区域)操作员有效地提取目标区域中的语义巷特征,并在目标区域和模型演员的未来互动中提取语义巷,这对未来的轨迹估计很大。 Ganet在所有公共文献(直到论文提交)中排名第一个,将其源代码排在第一位。
translated by 谷歌翻译
持续学习的现有工作(CL)的重点是减轻灾难性遗忘,即学习新任务时过去任务的模型绩效恶化。但是,CL系统的训练效率不足,这限制了CL系统在资源有限的方案下的现实应用。在这项工作中,我们提出了一个名为“稀疏持续学习”(SPARCL)的新颖框架,这是第一个利用稀疏性以使边缘设备上具有成本效益的持续学习的研究。 SPARCL通过三个方面的协同作用来实现训练加速度和准确性保护:体重稀疏性,数据效率和梯度稀疏性。具体而言,我们建议在整个CL过程中学习一个稀疏网络,动态数据删除(DDR),以删除信息较少的培训数据和动态梯度掩盖(DGM),以稀疏梯度更新。他们每个人不仅提高了效率,而且进一步减轻了灾难性的遗忘。 SPARCL始终提高现有最新CL方法(SOTA)CL方法的训练效率最多减少了训练失败,而且令人惊讶的是,SOTA的准确性最多最多提高了1.7%。 SPARCL还优于通过将SOTA稀疏训练方法适应CL设置的效率和准确性获得的竞争基线。我们还评估了SPARCL在真实手机上的有效性,进一步表明了我们方法的实际潜力。
translated by 谷歌翻译
准确的自我和相对状态估计是完成群体任务的关键前提,例如协作自主探索,目标跟踪,搜索和救援。本文提出了一种全面分散的状态估计方法,用于空中群体系统,其中每个无人机执行精确的自我状态估计,通过无线通信交换自我状态和相互观察信息,并估算相对状态(W.R.T.)(W.R.T.)无人机,全部实时,仅基于激光惯性测量。提出了一种基于3D激光雷达的新型无人机检测,识别和跟踪方法,以获得队友无人机的观察。然后,将相互观察测量与IMU和LIDAR测量紧密耦合,以实时和准确地估计自我状态和相对状态。广泛的现实世界实验显示了对复杂场景的广泛适应性,包括被GPS贬低的场景,摄影机的退化场景(漆黑的夜晚)或激光雷达(面对单个墙)。与运动捕获系统提供的地面真相相比,结果显示了厘米级的定位精度,该精度优于单个无人机系统的其他最先进的激光惯性射测。
translated by 谷歌翻译
模拟和混合信号(AMS)电路设计仍然依赖于人类设计专业知识。机器学习一直通过用人工智能代替人类的体验来协助电路设计自动化。本文介绍了标签,这是一种从利用文本,自我注意力和图形的布局中学习电路表示的新范式。嵌入网络模型在无手动标签的情况下学习空间信息。我们向AMS电路学习介绍文本嵌入和自我注意的机制。实验结果表明,具有工业罚款技术基准的实例之间的布局距离的能力。通过在案例研究中显示有限数据的其他三个学习任务的转移性,可以验证电路表示的有效性:布局匹配预测,线长度估计和净寄生电容预测。
translated by 谷歌翻译
最近对结构偏见进行了针对情感三胞胎提取(ASTE)的利用,并改善了性能。另一方面,人们认识到,明确纳入结构偏见会对效率产生负面影响,而预验证的语言模型(PLM)已经可以捕获隐式结构。因此,出现了一个自然的问题:在PLM的背景下,结构性偏见仍然是必要的吗?为了回答这个问题,我们建议通过使用适配器在PLM中整合结构偏置并使用便宜的计算相对位置结构来代替句法依赖性结构来解决效率问题。基准评估是在Semeval数据集上进行的。结果表明,我们提出的结构适配器对PLM有益,并在一系列强大的基准范围内实现最先进的性能,但具有光参数需求和延迟较低。同时,我们引起了人们的担忧,即当前的评估默认值为小规模的数据不足。因此,我们为ASTE发布了一个大型数据集。新数据集的结果暗示,结构适配器在大规模上自信地有效和有效。总体而言,我们得出一个结论,即即使使用PLM,结构偏见仍然是必要的。
translated by 谷歌翻译