This paper presents SimVTP: a Simple Video-Text Pretraining framework via masked autoencoders. We randomly mask out the spatial-temporal tubes of input video and the word tokens of input text and then feed them into a unified autencoder to reconstruct the missing pixels and words. Our SimVTP has several properties: 1) Thanks to the unified autoencoder, SimVTP reconstructs the masked signal of one modality with the help from another modality, which implicitly learns the cross-modal alignment between video tubes and text tokens. 2) SimVTP not only benefits from a high video masking ratio (e.g. 90%) due to the temporal redundancy of video, but also needs a high text masking ratio (e.g. 75%), which is much higher than BERT (e.g. 15%), to achieve optimal performance. This is because the aid of video modality makes text reconstruction less challenging, which thus needs a higher mask ratio to make the pretext harder for useful feature learning. 3) Equipping SimVTP with video-text contrastive learning (VTC) and video-text matching (VTM), which are two commonly used cross-modal training strategies, could further improve the transferable performance significantly. 4) SimVTP is dataefficent, e.g., pre-training only on 10% data of WebVid-2M, SimVTP achieves surprisingly good results (43.8 R@1) on MSRVTT, which is far above recent state-of-the-art methods pre-trained on both CC3M and WebVid-2M. We transfer our pre-trained model to various downstream tasks and achieve superior performance. The codes and models will be released at https://github.com/mayuelala/SimVTP.

构建一个通用视频语言模型，用于解决各种视频理解任务（例如，文本视频检索，视频问答）是对机器学习领域的开放挑战。为了实现这一目标，最近的尝试训练模型，通常由单峰和跨模式的特征编码器组成，并具有受监督或成对的对比度的预文本任务。尽管提供了有吸引力的通用性，但最终的模型必须在效率和性能之间妥协。我们认为这些缺陷是由它们的预训练策略\ Textemdash引起的，它们不能很好地对齐和融合不同方式的特征。然后，我们将三叶草（一种相关的视频预培训方法）介绍给一个通用的视频语言模型，该模型用于解决既不效率也不妥协的多个视频理解任务。它通过新的三模式比对预训练任务来改善跨模式特征对齐和融合。此外，我们建议通过合并蒙面样品的学习和新颖的成对排名损失来增强三模式对齐。三叶草表现出了出色的一般性。它在多个下游任务上建立了新的最新技术，包括零射击和微调设置的三个检索任务，以及八个视频问答任务。代码和预培训模型将在https://github.com/leeyn-43/clover上发布。

Video-language pre-training is crucial for learning powerful multi-modal representation. However, it typically requires a massive amount of computation. In this paper, we develop SMAUG, an efficient pre-training framework for video-language models. The foundation component in SMAUG is masked autoencoders. Different from prior works which only mask textual inputs, our masking strategy considers both visual and textual modalities, providing a better cross-modal alignment and saving more pre-training costs. On top of that, we introduce a space-time token sparsification module, which leverages context information to further select only "important" spatial regions and temporal frames for pre-training. Coupling all these designs allows our method to enjoy both competitive performances on text-to-video retrieval and video question answering tasks, and much less pre-training costs by 1.9X or more. For example, our SMAUG only needs about 50 NVIDIA A6000 GPU hours for pre-training to attain competitive performances on these two video-language tasks across six popular benchmarks.

We present a simple yet effective end-to-end Video-language Pre-training (VidLP) framework, Masked Contrastive Video-language Pretraining (MAC), for video-text retrieval tasks. Our MAC aims to reduce video representation's spatial and temporal redundancy in the VidLP model by a mask sampling mechanism to improve pre-training efficiency. Comparing conventional temporal sparse sampling, we propose to randomly mask a high ratio of spatial regions and only feed visible regions into the encoder as sparse spatial sampling. Similarly, we adopt the mask sampling technique for text inputs for consistency. Instead of blindly applying the mask-then-prediction paradigm from MAE, we propose a masked-then-alignment paradigm for efficient video-text alignment. The motivation is that video-text retrieval tasks rely on high-level alignment rather than low-level reconstruction, and multimodal alignment with masked modeling encourages the model to learn a robust and general multimodal representation from incomplete and unstable inputs. Coupling these designs enables efficient end-to-end pre-training: reduce FLOPs (60% off), accelerate pre-training (by 3x), and improve performance. Our MAC achieves state-of-the-art results on various video-text retrieval datasets, including MSR-VTT, DiDeMo, and ActivityNet. Our approach is omnivorous to input modalities. With minimal modifications, we achieve competitive results on image-text retrieval tasks.

图像和语言建模对于视觉前训练（VLP）至关重要，该培训旨在从大规模配对的图像文本数据中学习多模式表示。但是，我们观察到，大多数现有的VLP方法着重于建模图像和文本特征之间的相互作用，同时忽略图像和文本之间的信息差异，从而遭受焦点偏见。为了解决这个问题，我们提出了一个视觉语言掩盖自动编码器框架（VLMAE）。VLMAE采用视觉生成学习，促进该模型获得细粒度和公正的特征。与以前的作品不同，Vlmae注意图像中几乎所有关键的补丁，提供了更全面的理解。广泛的实验表明，VLMAE在各种视觉语言下游任务中取得更好的性能，包括视觉问答，即使有20％的预训练速度，图像文本检索和视觉接地也是如此。

在本文中，我们研究了如何在视觉和语言（V+L）表示学习中使用蒙版的信号建模。与其独立开发蒙面语言建模（MLM）和蒙面图像建模（MIM），我们建议建立关节蒙面的视觉和语言建模，其中一种模态的掩盖信号是在另一种方式的帮助下重建的。这是由图像文本配对数据的性质和文本传达几乎相同的信息但以不同格式传达的。在另一种模态下进行的一种模式的掩盖信号重建也可以隐式学习语言令牌和图像贴片之间的跨模式对齐。我们对各种V+L任务的实验表明，该建议的方法不仅可以通过使用大量数据来实现最先进的性能，而且还可以通过有限的培训数据的制度优于其他竞争对手。

视频语言（VIDL）建模的巨大挑战在于从图像/视频理解模型和下游Vidl数据中提取的固定视频表示之间的断开。最近的研究试图通过端到端培训来减轻这种断开连接。为了使其进行计算可行，先前的作品倾向于“想象”视频输入，即，将一些稀疏的采样帧馈送到2D CNN中，然后是简单的均值汇集或连接以获得整体视频表示。虽然实现了有希望的结果，但这种简单的方法可能会失去对于执行下游VIDL任务至关重要的时间信息。在这项工作中，我们呈现紫罗兰色，全新的视频语言变压器，采用视频变压器，明确地模拟视频输入的时间动态。此外，与以前的研究不同，发现视频输入上的预训练任务（例如，屏蔽帧建模）不是非常有效的，我们设计了一个新的预训练任务，屏蔽了视觉令牌建模（MVM），以获得更好的视频建模。具体地，原始视频帧修补程序将“令牌化”转换为离散的视觉令牌，目标是基于蒙面的贴片恢复原始的视觉令牌。综合分析展示了通过视频变压器和MVM显式时间建模的有效性。因此，紫罗兰在5个视频问题的回答任务和4个文本到视频检索任务中实现了新的最先进的性能。

Vision-Language Transformers can be learned without human labels (e.g. class labels, bounding boxes, etc). Existing work, whether explicitly utilizing bounding boxes or patches, assumes that the visual backbone must first be trained on ImageNet class prediction before being integrated into a multimodal linguistic pipeline. We show that this is not necessary and introduce a new model Vision-Language from Captions (VLC) built on top of Masked Auto-Encoders that does not require this supervision. In fact, in a head-to-head comparison between ViLT, the current state-of-the-art patch-based vision-language transformer which is pretrained with supervised object classification, and our model, VLC, we find that our approach 1. outperforms ViLT on standard benchmarks, 2. provides more interpretable and intuitive patch visualizations, and 3. is competitive with many larger models that utilize ROIs trained on annotated bounding-boxes.

本文介绍了Omnivl，这是一种新的基础模型，旨在使用一种通用体系结构来支持图像语言和视频语言任务。它为图像和视频输入采用了统一的基于变压器的视觉编码器，因此可以执行联合图像语言和视频语言预处理。我们首次证明了这样的范式受益于图像和视频任务，而不是传统的单向传输（例如，使用图像语言来帮助视频语言）。为此，我们提出了对图像语言和视频语言的脱钩关节预处理，以有效地将视觉模型分解为空间和时间维度，并在图像和视频任务上获得性能提升。此外，我们引入了一种新颖的统一视觉对比度（UNIVLC）损失，以利用图像文本，视频文本，图像标签（例如，图像分类），视频标签（例如，视频动作识别）在一起受到监督和吵闹的监督预处理数据都尽可能多地利用。无需额外的任务适配器，Omnivl可以同时支持仅视觉任务（例如，图像分类，视频操作识别），跨模式对齐任务（例如，图像/视频 - 文本检索）和多模式理解和生成任务（例如，图像/视频问答，字幕）。我们在各种下游任务上评估Omnivl，并以相似的模型大小和数据量表获得最新的或竞争结果。

蒙版的视觉建模（MVM）最近已被证明对视觉预训练有效。虽然在视频输入（例如，蒙版框架建模）上进行了类似的重建目标，在视频语言（VIDL）预训练中探索了类似的重建目标，但先前研究中的预提取的视频功能在预训练期间无法通过MVM进行完善，因此无法通过MVM进行完善为下游性能不满意。在这项工作中，我们系统地检查了MVM在VIDL学习的背景下的潜力。具体而言，我们的研究基于完全端到端的视频变压器（Violet），该视频变压器（Violet）减轻了固定视频表示与MVM培训之间的断开连接。总共探索了MVM的八个不同的重建目标，从低级像素值和定向梯度到高级深度图，光流，离散的视觉令牌和潜在的视觉特征。我们进行全面的实验，并就导致有效MVM培训的因素提供见解。从经验上讲，我们展示了通过MVM目标预先训练的紫罗兰色，可以在13个VIDL基准测试中取得显着改进，从视频问题回答，视频字幕到文本到视频检索等等。

The foundation models have recently shown excellent performance on a variety of downstream tasks in computer vision. However, most existing vision foundation models simply focus on image-level pretraining and adpation, which are limited for dynamic and complex video-level understanding tasks. To fill the gap, we present general video foundation models, InternVideo, by taking advantage of both generative and discriminative self-supervised video learning. Specifically, InternVideo efficiently explores masked video modeling and video-language contrastive learning as the pretraining objectives, and selectively coordinates video representations of these two complementary frameworks in a learnable manner to boost various video applications. Without bells and whistles, InternVideo achieves state-of-the-art performance on 39 video datasets from extensive tasks including video action recognition/detection, video-language alignment, and open-world video applications. Especially, our methods can obtain 91.1% and 77.2% top-1 accuracy on the challenging Kinetics-400 and Something-Something V2 benchmarks, respectively. All of these results effectively show the generality of our InternVideo for video understanding. The code will be released at https://github.com/OpenGVLab/InternVideo .

视频文本预训练（VTP）旨在从大规模的网络视频中学习可转移的代表。迄今为止，几乎所有现有的VTP方法都仅限于基于检索的下游任务，例如视频检索，而它们在基于本地化的任务（例如时间基础）上的转移潜力不足。在本文中，我们实验分析并证明了当前VTP方法与本地化任务的不相容性，并提出了一种新颖的面向定位的视频文本预训练框架，称为LocvTP。具体而言，我们执行细粒对比度对准作为通过剪贴字对数发现方案对粗粒粒度的补充。为了进一步增强学习功能的时间推理能力，我们提出了一个上下文投影头和暂时意识的对比损失，以感知上下文关系。对六个数据集的四个下游任务进行的广泛实验表明，我们的LOCVTP在基于检索和基于本地化的任务上都达到了最先进的性能。此外，我们进行了全面的消融研究和彻底的分析，以探索最佳的模型设计和培训策略。

Video-language pre-training has advanced the performance of various downstream video-language tasks. However, most previous methods directly inherit or adapt typical image-language pre-training paradigms to video-language pre-training, thus not fully exploiting the unique characteristic of video, i.e., temporal. In this paper, we propose a Hierarchical Temporal-Aware video-language pre-training framework, HiTeA, with two novel pre-training tasks for modeling cross-modal alignment between moments and texts as well as the temporal relations of video-text pairs. Specifically, we propose a cross-modal moment exploration task to explore moments in videos, which results in detailed video moment representation. Besides, the inherent temporal relations are captured by aligning video-text pairs as a whole in different time resolutions with multi-modal temporal relation exploration task. Furthermore, we introduce the shuffling test to evaluate the temporal reliance of datasets and video-language pre-training models. We achieve state-of-the-art results on 15 well-established video-language understanding and generation tasks, especially on temporal-oriented datasets (e.g., SSv2-Template and SSv2-Label) with 8.6% and 11.1% improvement respectively. HiTeA also demonstrates strong generalization ability when directly transferred to downstream tasks in a zero-shot manner. Models and demo will be available on ModelScope.

随着变压器的发展，近年来预先训练的模型已经以突破性的步伐发展。他们在自然语言处理（NLP）和计算机视觉（CV）中主导了主流技术。如何将预训练适应视觉和语言（V-L）学习和改善下游任务绩效成为多模式学习的重点。在本文中，我们回顾了视力语言预训练模型（VL-PTMS）的最新进展。作为核心内容，我们首先简要介绍了几种方法，将原始图像和文本编码为单模式嵌入在预训练之前。然后，我们在建模文本和图像表示之间的相互作用时深入研究VL-PTM的主流体系结构。我们进一步提出了广泛使用的预训练任务，然后我们介绍了一些常见的下游任务。我们终于结束了本文，并提出了一些有前途的研究方向。我们的调查旨在为研究人员提供合成和指向相关研究的指针。

Learning fine-grained interplay between vision and language allows to a more accurate understanding for VisionLanguage tasks. However, it remains challenging to extract key image regions according to the texts for semantic alignments. Most existing works are either limited by textagnostic and redundant regions obtained with the frozen detectors, or failing to scale further due to its heavy reliance on scarce grounding (gold) data to pre-train detectors. To solve these problems, we propose Self-Locator Aided Network (SLAN) for cross-modal understanding tasks without any extra gold data. SLAN consists of a region filter and a region adaptor to localize regions of interest conditioned on different texts. By aggregating cross-modal information, the region filter selects key regions and the region adaptor updates their coordinates with text guidance. With detailed region-word alignments, SLAN can be easily generalized to many downstream tasks. It achieves fairly competitive results on five cross-modal understanding tasks (e.g., 85.7% and 69.2% on COCO image-to-text and text-to-image retrieval, surpassing previous SOTA methods). SLAN also demonstrates strong zero-shot and fine-tuned transferability to two localization tasks.

近年来，统一的视觉语言框架已经大大提高，其中大多数采用编码器架构将图像文本任务统一为序列到序列的生成。但是，现有的视频语言（VIDL）模型仍需要在每个任务的模型体系结构和培训目标中进行特定于任务的设计。在这项工作中，我们探索了一个统一的VIDL框架薰衣草，其中蒙版语言建模（MLM）用作所有前训练和下游任务的常见接口。这样的统一导致了简化的模型体系结构，在多模式编码器之上，只需要一个轻巧的MLM头，而不是具有更多参数的解码器。令人惊讶的是，实验结果表明，这个统一的框架在14个VIDL基准测试中实现了竞争性能，涵盖了视频问答，文本到视频检索和视频字幕。广泛的分析进一步证明了薰衣草比现有VIDL方法的优势：（i）在多任务列出时仅使用一组参数值支持所有下游任务； （ii）对各种下游任务的几乎没有概括； （iii）在视频问题回答任务上启用零射门评估。代码可从https://github.com/microsoft/lavender获得。

在过去的几年中，训练前模型的出现将单峰领域（例如计算机视觉（CV）和自然语言处理（NLP））带到了一个新时代。实质性的作品表明它们对下游大学任务有益，并避免从头开始训练新的模型。那么，此类预训练的模型可以应用于多模式任务吗？研究人员探索了这个问题并取得了重大进展。本文调查了视觉预训练（VLP）的最新进展和新的前沿，包括图像文本和视频文本预训练。为了使读者更好地掌握VLP，我们首先从五个方面回顾了其最新进展：功能提取，模型体系结构，培训预训练目标，预训练数据集和下游任务。然后，我们详细概述了特定的VLP模型。最后，我们讨论了VLP中的新边界。据我们所知，这是对VLP的首次调查。我们希望这项调查能够阐明VLP领域的未来研究。

通常需要在大型数据集上进行预训练的视频变压器，以在相对较小的数据集上实现首要性能。在本文中，我们表明视频蒙面的自动编码器（Videomae）是用于自我监督视频预训练（SSVP）的数据效率学习者。我们的启发受到了最近的ImageMae的启发，并提出了具有极高比例的定制视频管掩蔽。这种简单的设计使视频重建成为更具挑战性的自我判断任务，从而鼓励在此预训练过程中提取更有效的视频表示。我们在SSVP上获得了三个重要发现：（1）屏蔽比的比例极高（即90％至95％）仍然可以产生良好的视频性能。在时间上冗余的视频内容比图像更高的掩蔽率。 （2）视频在很小的数据集（即3K-4K视频）上取得了令人印象深刻的结果，而无需使用任何额外的数据。 （3）视频表明，数据质量比SSVP的数据数量更重要。在培训和目标数据集之间的域转移是一个重要问题。值得注意的是，我们与香草VIT的视频在动力学400上可以达到85.8％，在不使用任何额外数据的情况下，在HMDB51上的V2上有75.3％，UCF101的某些东西为75.3％，在UCF101上获得90.8％，HMDB51上的90.8％和61.1％。代码可从https://github.com/mcg-nju/videomae获得。

探索大规模预处理的基础模型对计算机视觉具有重大兴趣，因为这些模型可以快速转移到许多下游任务中。本文介绍了对比字幕（COCA），这是一种极简主义的设计，旨在为图像文本编码器编码器基础模型预算与对比度损失和字幕损失，从而从剪辑和诸如simvlm之类的生成方法之类的对比方法中包含模型能力。与所有解码器层都参与编码器输出的标准编码器 - 模块变压器相反，可口可乐省略了解码器层的上半部分的交叉注意，以编码单峰文本表示，并串联到剩余的解码器层，这些解码器与图像编码器相交的解码器层多模式图像文本表示。除了对多模态解码器输出的字幕损失外，我们还应用了单峰图像和文本嵌入之间的对比损失，该输出可以预测文本令牌自动加压。通过共享相同的计算图，可以用最小的开销有效地计算两个培训目标。可口可乐是端到端和从头开始的网络尺度alt-text数据和带注释的图像，通过将所有标签视为文本，无缝地统一自然语言监督以进行表示。从经验上讲，可口可乐通过零拍传输或在广泛的下游任务上进行零摄像转移或最少的特定任务适应，跨越视觉识别（Imagenet，Kinetics-400/600/700，瞬间， ），交叉模式检索（MSCOCO，FLICKR30K，MSR-VTT），多模式理解（VQA，SNLI-VE，NLVR2）和图像字幕（MSCOCO，NOCAPS）。值得注意的是，在Imagenet分类方面，COCA获得了86.3％的TOP-1准确性，带有冷冻编码器和学习的分类头90.6％，以及带有填充编码器的Imagenet上的新最先进的91.0％Top-1 Top-1精度。

The canonical approach to video-and-language learning (e.g., video question answering) dictates a neural model to learn from offline-extracted dense video features from vision models and text features from language models. These feature extractors are trained independently and usually on tasks different from the target domains, rendering these fixed features sub-optimal for downstream tasks. Moreover, due to the high computational overload of dense video features, it is often difficult (or infeasible) to plug feature extractors directly into existing approaches for easy finetuning. To provide a remedy to this dilemma, we propose a generic framework CLIPBERT that enables affordable endto-end learning for video-and-language tasks, by employing sparse sampling, where only a single or a few sparsely sampled short clips from a video are used at each training step. Experiments on text-to-video retrieval and video question answering on six datasets demonstrate that CLIP-BERT outperforms (or is on par with) existing methods that exploit full-length videos, suggesting that end-to-end learning with just a few sparsely sampled clips is often more accurate than using densely extracted offline features from full-length videos, proving the proverbial less-is-more principle. Videos in the datasets are from considerably different domains and lengths, ranging from 3-second genericdomain GIF videos to 180-second YouTube human activity videos, showing the generalization ability of our approach. Comprehensive ablation studies and thorough analyses are provided to dissect what factors lead to this success. Our code is publicly available. 1 * Equal contribution.