注意机制对研究界提出了重大兴趣，因为他们承诺改善神经网络架构的表现。但是，在任何特定的问题中，我们仍然缺乏主要的方法来选择导致保证改进的具体机制和超参数。最近，已经提出了自我关注并广泛用于变压器 - 类似的架构中，导致某些应用中的重大突破。在这项工作中，我们专注于两种形式的注意机制：注意模块和自我关注。注意模块用于重新重量每个层输入张量的特征。不同的模块具有不同的方法，可以在完全连接或卷积层中执行此重复。研究的注意力模型是完全模块化的，在这项工作中，它们将与流行的Reset架构一起使用。自我关注，最初在自然语言处理领域提出，可以将所有项目与输入序列中的所有项目相关联。自我关注在计算机视觉中越来越受欢迎，其中有时与卷积层相结合，尽管最近的一些架构与卷曲完全消失。在这项工作中，我们研究并执行了在特定计算机视觉任务中许多不同关注机制的客观的比较，在广泛使用的皮肤癌MNIST数据集中的样本分类。结果表明，关注模块有时会改善卷积神经网络架构的性能，也是这种改进虽然明显且统计学意义，但在不同的环境中并不一致。另一方面，通过自我关注机制获得的结果表明了一致和显着的改进，即使在具有减少数量的参数的架构中，也可以实现最佳结果。

While the Transformer architecture has become the de-facto standard for natural language processing tasks, its applications to computer vision remain limited. In vision, attention is either applied in conjunction with convolutional networks, or used to replace certain components of convolutional networks while keeping their overall structure in place. We show that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts of data and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.), Vision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train. 1

Vision Transformers have shown great promise recently for many vision tasks due to the insightful architecture design and attention mechanism. By revisiting the self-attention responses in Transformers, we empirically observe two interesting issues. First, Vision Transformers present a queryirrelevant behavior at deep layers, where the attention maps exhibit nearly consistent contexts in global scope, regardless of the query patch position (also head-irrelevant). Second, the attention maps are intrinsically sparse, few tokens dominate the attention weights; introducing the knowledge from ConvNets would largely smooth the attention and enhance the performance. Motivated by above observations, we generalize self-attention formulation to abstract a queryirrelevant global context directly and further integrate the global context into convolutions. The resulting model, a Fully Convolutional Vision Transformer (i.e., FCViT), purely consists of convolutional layers and firmly inherits the merits of both attention mechanism and convolutions, including dynamic property, weight sharing, and short- and long-range feature modeling, etc. Experimental results demonstrate the effectiveness of FCViT. With less than 14M parameters, our FCViT-S12 outperforms related work ResT-Lite by 3.7% top1 accuracy on ImageNet-1K. When scaling FCViT to larger models, we still perform better than previous state-of-the-art ConvNeXt with even fewer parameters. FCViT-based models also demonstrate promising transferability to downstream tasks, like object detection, instance segmentation, and semantic segmentation. Codes and models are made available at: https://github.com/ma-xu/FCViT.

We design a family of image classification architectures that optimize the trade-off between accuracy and efficiency in a high-speed regime. Our work exploits recent findings in attention-based architectures, which are competitive on highly parallel processing hardware. We revisit principles from the extensive literature on convolutional neural networks to apply them to transformers, in particular activation maps with decreasing resolutions. We also introduce the attention bias, a new way to integrate positional information in vision transformers.As a result, we propose LeVIT: a hybrid neural network for fast inference image classification. We consider different measures of efficiency on different hardware platforms, so as to best reflect a wide range of application scenarios. Our extensive experiments empirically validate our technical choices and show they are suitable to most architectures. Overall, LeViT significantly outperforms existing convnets and vision transformers with respect to the speed/accuracy tradeoff. For example, at 80% ImageNet top-1 accuracy, LeViT is 5 times faster than EfficientNet on CPU. We release the code at https: //github.com/facebookresearch/LeViT.

随着变压器作为语言处理的标准及其在计算机视觉方面的进步，参数大小和培训数据的数量相应地增长。许多人开始相信，因此，变形金刚不适合少量数据。这种趋势引起了人们的关注，例如：某些科学领域中数据的可用性有限，并且排除了该领域研究资源有限的人。在本文中，我们旨在通过引入紧凑型变压器来提出一种小规模学习的方法。我们首次表明，具有正确的尺寸，卷积令牌化，变压器可以避免在小数据集上过度拟合和优于最先进的CNN。我们的模型在模型大小方面具有灵活性，并且在获得竞争成果的同时，参数可能仅为0.28亿。当在CIFAR-10上训练Cifar-10，只有370万参数训练时，我们的最佳模型可以达到98％的准确性，这是与以前的基于变形金刚的模型相比，数据效率的显着提高，比其他变压器小于10倍，并且是15％的大小。在实现类似性能的同时，重新NET50。 CCT还表现优于许多基于CNN的现代方法，甚至超过一些基于NAS的方法。此外，我们在Flowers-102上获得了新的SOTA，具有99.76％的TOP-1准确性，并改善了Imagenet上现有基线（82.71％精度，具有29％的VIT参数）以及NLP任务。我们针对变压器的简单而紧凑的设计使它们更可行，可以为那些计算资源和/或处理小型数据集的人学习，同时扩展了在数据高效变压器中的现有研究工作。我们的代码和预培训模型可在https://github.com/shi-labs/compact-transformers上公开获得。

哥内克人Sentinel Imagery的纯粹卷的可用性为使用深度学习的大尺度创造了新的土地利用陆地覆盖（Lulc）映射的机会。虽然在这种大型数据集上培训是一个非琐碎的任务。在这项工作中，我们试验Lulc Image分类和基准不同最先进模型的Bigearthnet数据集，包括卷积神经网络，多层感知，视觉变压器，高效导通和宽残余网络（WRN）架构。我们的目标是利用分类准确性，培训时间和推理率。我们提出了一种基于用于网络深度，宽度和输入数据分辨率的WRNS复合缩放的高效导通的框架，以有效地训练和测试不同的模型设置。我们设计一种新颖的缩放WRN架构，增强了有效的通道注意力机制。我们提出的轻量级模型具有较小的培训参数，实现所有19个LULC类的平均F分类准确度达到4.5％，并且验证了我们使用的resnet50最先进的模型速度快两倍作为基线。我们提供超过50种培训的型号，以及我们在多个GPU节点上分布式培训的代码。

视觉变压器（VIT）用作强大的视觉模型。与卷积神经网络不同，在前几年主导视觉研究，视觉变压器享有捕获数据中的远程依赖性的能力。尽管如此，任何变压器架构的组成部分，自我关注机制都存在高延迟和低效的内存利用，使其不太适合高分辨率输入图像。为了缓解这些缺点，分层视觉模型在非交错的窗口上局部使用自我关注。这种放松会降低输入尺寸的复杂性;但是，它限制了横窗相互作用，损害了模型性能。在本文中，我们提出了一种新的班次不变的本地注意层，称为查询和参加（QNA），其以重叠的方式聚集在本地输入，非常类似于卷积。 QNA背后的关键想法是介绍学习的查询，这允许快速高效地实现。我们通过将其纳入分层视觉变压器模型来验证我们的层的有效性。我们展示了速度和内存复杂性的改进，同时实现了与最先进的模型的可比准确性。最后，我们的图层尺寸尤其良好，窗口大小，需要高于X10的内存，而不是比现有方法更快。

人类自然有效地在复杂的场景中找到突出区域。通过这种观察的动机，引入了计算机视觉中的注意力机制，目的是模仿人类视觉系统的这一方面。这种注意机制可以基于输入图像的特征被视为动态权重调整过程。注意机制在许多视觉任务中取得了巨大的成功，包括图像分类，对象检测，语义分割，视频理解，图像生成，3D视觉，多模态任务和自我监督的学习。在本调查中，我们对计算机愿景中的各种关注机制进行了全面的审查，并根据渠道注意，空间关注，暂时关注和分支注意力进行分类。相关的存储库https：//github.com/menghaoguo/awesome-vision-tions致力于收集相关的工作。我们还建议了未来的注意机制研究方向。

We present in this paper a new architecture, named Convolutional vision Transformer (CvT), that improves Vision Transformer (ViT) in performance and efficiency by introducing convolutions into ViT to yield the best of both designs. This is accomplished through two primary modifications: a hierarchy of Transformers containing a new convolutional token embedding, and a convolutional Transformer block leveraging a convolutional projection. These changes introduce desirable properties of convolutional neural networks (CNNs) to the ViT architecture (i.e. shift, scale, and distortion invariance) while maintaining the merits of Transformers (i.e. dynamic attention, global context, and better generalization). We validate CvT by conducting extensive experiments, showing that this approach achieves state-of-the-art performance over other Vision Transformers and ResNets on ImageNet-1k, with fewer parameters and lower FLOPs. In addition, performance gains are maintained when pretrained on larger datasets (e.g. ImageNet-22k) and fine-tuned to downstream tasks. Pretrained on ImageNet-22k, our CvT-W24 obtains a top-1 accuracy of 87.7% on the ImageNet-1k val set. Finally, our results show that the positional encoding, a crucial component in existing Vision Transformers, can be safely removed in our model, simplifying the design for higher resolution vision tasks. Code will be released at https: //github.com/leoxiaobin/CvT.

Transformer is a new kind of neural architecture which encodes the input data as powerful features via the attention mechanism. Basically, the visual transformers first divide the input images into several local patches and then calculate both representations and their relationship. Since natural images are of high complexity with abundant detail and color information, the granularity of the patch dividing is not fine enough for excavating features of objects in different scales and locations. In this paper, we point out that the attention inside these local patches are also essential for building visual transformers with high performance and we explore a new architecture, namely, Transformer iN Transformer (TNT). Specifically, we regard the local patches (e.g., 16×16) as "visual sentences" and present to further divide them into smaller patches (e.g., 4×4) as "visual words". The attention of each word will be calculated with other words in the given visual sentence with negligible computational costs. Features of both words and sentences will be aggregated to enhance the representation ability. Experiments on several benchmarks demonstrate the effectiveness of the proposed TNT architecture, e.g., we achieve an 81.5% top-1 accuracy on the ImageNet, which is about 1.7% higher than that of the state-of-the-art visual transformer with similar computational cost.

在本文中，我们询问视觉变形金刚（VIT）是否可以作为改善机器学习模型对抗逃避攻击的对抗性鲁棒性的基础结构。尽管较早的作品集中在改善卷积神经网络上，但我们表明VIT也非常适合对抗训练以实现竞争性能。我们使用自定义的对抗训练配方实现了这一目标，该配方是在Imagenet数据集的一部分上使用严格的消融研究发现的。与卷积相比，VIT的规范培训配方建议强大的数据增强，部分是为了补偿注意力模块的视力归纳偏置。我们表明，该食谱在用于对抗训练时可实现次优性能。相比之下，我们发现省略所有重型数据增强，并添加一些额外的零件（$ \ varepsilon $ -Warmup和更大的重量衰减），从而大大提高了健壮的Vits的性能。我们表明，我们的配方在完整的Imagenet-1k上概括了不同类别的VIT体系结构和大规模模型。此外，调查了模型鲁棒性的原因，我们表明，在使用我们的食谱时，在训练过程中产生强烈的攻击更加容易，这会在测试时提高鲁棒性。最后，我们通过提出一种量化对抗性扰动的语义性质并强调其与模型的鲁棒性的相关性来进一步研究对抗训练的结果。总体而言，我们建议社区应避免将VIT的规范培训食谱转换为在对抗培训的背景下进行强大的培训和重新思考常见的培训选择。

视觉变压器由于能够捕获图像中的长期依赖性的能力而成功地应用于图像识别任务。但是，变压器与现有卷积神经网络（CNN）之间的性能和计算成本仍然存在差距。在本文中，我们旨在解决此问题，并开发一个网络，该网络不仅可以超越规范变压器，而且可以超越高性能卷积模型。我们通过利用变压器来捕获长期依赖性和CNN来建模本地特征，从而提出了一个新的基于变压器的混合网络。此外，我们将其扩展为获得一个称为CMT的模型家族，比以前的基于卷积和基于变压器的模型获得了更好的准确性和效率。特别是，我们的CMT-S在ImageNet上获得了83.5％的TOP-1精度，而在拖鞋上的拖曳率分别比现有的DEIT和EficitiveNet小14倍和2倍。拟议的CMT-S还可以很好地概括CIFAR10（99.2％），CIFAR100（91.7％），花（98.7％）以及其他具有挑战性的视觉数据集，例如可可（44.3％地图），计算成本较小。

过去一年目睹了将变压器模块应用于视力问题的快速发展。虽然一些研究人员已经证明，基于变压器的模型享有有利的拟合数据能力，但仍然越来越多的证据，表明这些模型尤其在训练数据受到限制时遭受过度拟合。本文通过执行逐步操作来提供实证研究，逐步运输基于变压器的模型到基于卷积的模型。我们在过渡过程中获得的结果为改善视觉识别提供了有用的消息。基于这些观察，我们提出了一个名为VIRFormer的新架构，该体系结构从“视觉友好的变压器”中缩写。具有相同的计算复杂度，在想象集分类精度方面，VISFormer占据了基于变压器的基于卷积的模型，并且当模型复杂性较低或训练集较小时，优势变得更加重要。代码可在https://github.com/danczs/visformer中找到。

在本文中，我们介绍了一种新颖的视觉表示学习，它依赖于少数自适应地学习令牌，并且适用于图像和视频理解任务。而不是依靠手工设计的分割策略来获得视觉令牌并处理大量密集采样的补丁进行关注，我们的方法学会在视觉数据中挖掘重要令牌。这导致有效且有效地找到一些重要的视觉令牌，并且可以在这些令牌之间进行成像注意，在更长的视频的时间范围内，或图像中的空间内容。我们的实验表现出对图像和视频识别任务的几个具有挑战性的基准的强烈性能。重要的是，由于我们的令牌适应性，我们在显着减少的计算金额下实现竞争结果。在计算上更有效的同时，我们获得了对想象成的最先进结果的可比结果。我们在多个视频数据集中建立新的最先进的，包括动力学-400，动力学-600，Charades和Avid。代码可在：https://github.com/google-research/scenic/tree/main/scenic/projects/token_learner

由于2017年介绍了变压器架构，因此许多尝试将自我关注范例带入计算机愿景领域。在本文中，我们提出了一种新颖的自我关注模块，可以很容易地集成在几乎每个卷积神经网络中，专门为计算机视觉设计，LHC：本地（多）头通道（自我关注）。 LHC是基于两个主要思想：首先，我们认为在电脑视觉中利用自我关注范式的最佳方式是渠道明智的应用而不是更探索的空间关注，并且卷积不会被引起的注意力替换经常性网络在NLP中;其次，局部方法有可能更好地克服卷积的局限性而不是全球关注。通过LHC-Net，我们设法在着名的FER2013数据集中实现了新的艺术状态，与先前的SOTA相比，在计算成本方面的复杂性和对“宿主”架构的复杂性显着和影响。

Vision Transformer (ViT) extends the application range of transformers from language processing to computer vision tasks as being an alternative architecture against the existing convolutional neural networks (CNN). Since the transformer-based architecture has been innovative for computer vision modeling, the design convention towards an effective architecture has been less studied yet. From the successful design principles of CNN, we investigate the role of spatial dimension conversion and its effectiveness on transformer-based architecture. We particularly attend to the dimension reduction principle of CNNs; as the depth increases, a conventional CNN increases channel dimension and decreases spatial dimensions. We empirically show that such a spatial dimension reduction is beneficial to a transformer architecture as well, and propose a novel Pooling-based Vision Transformer (PiT) upon the original ViT model. We show that PiT achieves the improved model capability and generalization performance against ViT. Throughout the extensive experiments, we further show PiT outperforms the baseline on several tasks such as image classification, object detection, and robustness evaluation. Source codes and ImageNet models are available at https://github.com/naver-ai/pit.

Convolutional Neural Networks (CNNs) are the go-to model for computer vision. Recently, attention-based networks, such as the Vision Transformer, have also become popular. In this paper we show that while convolutions and attention are both sufficient for good performance, neither of them are necessary. We present MLP-Mixer, an architecture based exclusively on multi-layer perceptrons (MLPs). MLP-Mixer contains two types of layers: one with MLPs applied independently to image patches (i.e. "mixing" the per-location features), and one with MLPs applied across patches (i.e. "mixing" spatial information). When trained on large datasets, or with modern regularization schemes, MLP-Mixer attains competitive scores on image classification benchmarks, with pre-training and inference cost comparable to state-of-the-art models. We hope that these results spark further research beyond the realms of well established CNNs and Transformers. 1

Attention-based neural networks, such as Transformers, have become ubiquitous in numerous applications, including computer vision, natural language processing, and time-series analysis. In all kinds of attention networks, the attention maps are crucial as they encode semantic dependencies between input tokens. However, most existing attention networks perform modeling or reasoning based on representations, wherein the attention maps of different layers are learned separately without explicit interactions. In this paper, we propose a novel and generic evolving attention mechanism, which directly models the evolution of inter-token relationships through a chain of residual convolutional modules. The major motivations are twofold. On the one hand, the attention maps in different layers share transferable knowledge, thus adding a residual connection can facilitate the information flow of inter-token relationships across layers. On the other hand, there is naturally an evolutionary trend among attention maps at different abstraction levels, so it is beneficial to exploit a dedicated convolution-based module to capture this process. Equipped with the proposed mechanism, the convolution-enhanced evolving attention networks achieve superior performance in various applications, including time-series representation, natural language understanding, machine translation, and image classification. Especially on time-series representation tasks, Evolving Attention-enhanced Dilated Convolutional (EA-DC-) Transformer outperforms state-of-the-art models significantly, achieving an average of 17% improvement compared to the best SOTA. To the best of our knowledge, this is the first work that explicitly models the layer-wise evolution of attention maps. Our implementation is available at https://github.com/pkuyym/EvolvingAttention

There still remains an extreme performance gap between Vision Transformers (ViTs) and Convolutional Neural Networks (CNNs) when training from scratch on small datasets, which is concluded to the lack of inductive bias. In this paper, we further consider this problem and point out two weaknesses of ViTs in inductive biases, that is, the spatial relevance and diverse channel representation. First, on spatial aspect, objects are locally compact and relevant, thus fine-grained feature needs to be extracted from a token and its neighbors. While the lack of data hinders ViTs to attend the spatial relevance. Second, on channel aspect, representation exhibits diversity on different channels. But the scarce data can not enable ViTs to learn strong enough representation for accurate recognition. To this end, we propose Dynamic Hybrid Vision Transformer (DHVT) as the solution to enhance the two inductive biases. On spatial aspect, we adopt a hybrid structure, in which convolution is integrated into patch embedding and multi-layer perceptron module, forcing the model to capture the token features as well as their neighboring features. On channel aspect, we introduce a dynamic feature aggregation module in MLP and a brand new "head token" design in multi-head self-attention module to help re-calibrate channel representation and make different channel group representation interacts with each other. The fusion of weak channel representation forms a strong enough representation for classification. With this design, we successfully eliminate the performance gap between CNNs and ViTs, and our DHVT achieves a series of state-of-the-art performance with a lightweight model, 85.68% on CIFAR-100 with 22.8M parameters, 82.3% on ImageNet-1K with 24.0M parameters. Code is available at https://github.com/ArieSeirack/DHVT.

最近的几项研究表明，基于关注的网络，如视觉变压器（VIV），可以在几个计算机视觉任务上倾斜卷积神经网络（CNNS）而不使用卷积层。这自然导致以下问题：可以自我关注的Vit表达任何卷积操作吗？在这项工作中，我们证明了一种具有图像贴片的单个VIT层，因为输入可以建设性地执行任何卷积操作，其中多主题注意机制和相对位置编码起到基本角色。我们进一步提供了视觉变压器的头部数量的下限，以表达CNN。对应于我们的分析，实验结果表明，我们证据的建设可以帮助将卷积偏差注入变压器，并显着提高vit的低数据制度的性能。