在本报告中，我们建议针对四个EGO4D挑战任务，包括自然语言查询（NLQ），MOMMER QUERY（MQ），对象状态变更分类（OSCC），以及PNR定位（PNR）。尤其是，我们将最近发布的EGO4D数据集\ cite {grauman2021ego4d}从预处理数据集，预处理目标和开发集中从egecentric vlp中提升。基于上述三个设计，我们开发了一个验证的视频语言模型，该模型能够将其以自我为中心的视频文本表示或仅视频表示形式转移到几个视频下游任务中。我们的Egentric VLP在NLQ上实现10.46r@1&iou @0.3，MQ上的10.33地图，OSCC上的74％ACC，PNR上的0.67秒错误。该代码可在https://github.com/showlab/egovlp上找到。

在本报告中，我们为Epic-kitchens-100多实体检索（miR）挑战提出了一个基于视频的预处理（VLP）解决方案\ cite {kevin202222222egovlp}。尤其是，我们将最近发布的EGO4D数据集\ cite {grauman2021ego4d}从预处理数据集，预处理目标和开发集中从egecentric vlp中提升。基于上述三个设计，我们开发了一个预验证的视频语言模型，该模型能够将其自我为中心的视频文本表示为mir基准。此外，我们设计了一种自适应多构度最大损失，以有效地微调模型并为可靠的推理配备双重效果技术。我们最好的单个模型在挑战测试集上获得了强劲的性能，其中47.39％的地图和61.44％的NDCG。该代码可在https://github.com/showlab/egovlp上找到。

预先培训用于学习可转让的视频文本表示的模型，以近年来引起了很多关注。以前的主导作品主要采用两个独立的编码器来有效检索，但忽略视频和文本之间的本地关联。另一种研究使用联合编码器与文本交互视频，但是由于每个文本视频对需要馈送到模型中的低效率。在这项工作中，我们能够通过新颖的借口任务进行微粒视频文本交互，以便通过新颖的借口任务进行检索，称为多项选择题（MCQ），其中参数模块BridgeFormer培训以接受由此构建的“问题”。文本功能通过诉诸视频功能。具体来说，我们利用了文本的丰富语义（即，名词和动词）来构建问题，可以培训视频编码器以捕获更多区域内容和时间动态。以问题和答案的形式，可以正确建立本地视频文本功能之间的语义关联。 BridgeFormer能够删除下游检索，只有两个编码器渲染高效且灵活的模型。我们的方法在具有不同实验设置（即零拍摄和微调）的五个数据集中，在五个数据集中优于最先进的方法，包括不同的实验设置（即零拍摄和微调），包括HOWTO100M（一百万个视频）。我们进一步开展零射击动作识别，可以作为视频到文本检索，我们的方法也显着超越了其对应物。作为额外的好处，我们的方法在单模下游任务中实现了竞争力，在单模下游任务上具有更短的预训练视频，例如，使用线性评估的动作识别。

We introduce LaViLa, a new approach to learning video-language representations by leveraging Large Language Models (LLMs). We repurpose pre-trained LLMs to be conditioned on visual input, and finetune them to create automatic video narrators. Our auto-generated narrations offer a number of advantages, including dense coverage of long videos, better temporal synchronization of the visual information and text, and much higher diversity of text. The video-text embedding learned contrastively with these additional auto-generated narrations outperforms the previous state-of-the-art on multiple first-person and third-person video tasks, both in zero-shot and finetuned setups. Most notably, LaViLa obtains an absolute gain of 10.1% on EGTEA classification and 5.9% Epic-Kitchens-100 multi-instance retrieval benchmarks. Furthermore, LaViLa trained with only half the narrations from the Ego4D dataset outperforms baseline models trained on the full set, and shows positive scaling behavior on increasing pre-training data and model size.

最近，通过引入大规模的数据集和强大的变压器网络，视频预培训表明尤其是检索的巨大成功。然而，现有的视频语言变压器模型没有明确细粒度的语义对齐。在这项工作中，我们呈现了对象感知的变换器，以对象为中心的方法，该对象方法扩展了视频语言变压器来合并对象表示。关键的想法是利用边界框和对象标签来指导培训过程。我们在四个广泛使用的基准测试中评估了我们的三个标准子任务的模型。我们还提供了深入的分析和详细消融关于所提出的方法。我们在考虑的所有任务和数据集中表现出清晰的性能，展示将对象表示的模型中的型号集成到视频架构中。代码将以\ URL {https://github.com/fingerrec/oa -transformer}释放。

The foundation models have recently shown excellent performance on a variety of downstream tasks in computer vision. However, most existing vision foundation models simply focus on image-level pretraining and adpation, which are limited for dynamic and complex video-level understanding tasks. To fill the gap, we present general video foundation models, InternVideo, by taking advantage of both generative and discriminative self-supervised video learning. Specifically, InternVideo efficiently explores masked video modeling and video-language contrastive learning as the pretraining objectives, and selectively coordinates video representations of these two complementary frameworks in a learnable manner to boost various video applications. Without bells and whistles, InternVideo achieves state-of-the-art performance on 39 video datasets from extensive tasks including video action recognition/detection, video-language alignment, and open-world video applications. Especially, our methods can obtain 91.1% and 77.2% top-1 accuracy on the challenging Kinetics-400 and Something-Something V2 benchmarks, respectively. All of these results effectively show the generality of our InternVideo for video understanding. The code will be released at https://github.com/OpenGVLab/InternVideo .

视频文本检索一直是多模式研究中的至关重要和基本任务。大型多模式对比预训练的发展，视频文本检索的开发已大大促进，这主要侧重于粗粒或细粒对比。然而，在先前的研究中很少探索过跨粒度的对比，这是粗粒表示和细粒度表示之间的对比。与细粒度或粗粒的对比相比，交叉粒度对比度计算了粗粒粒度特征与每个细粒特征之间的相关性，并且能够过滤出不必要的细颗粒特征，这些特征由粗粒度的特征引导相似性计算，从而提高了检索的准确性。为此，本文提出了一种新型的多透明对比模型，即X-CLIP，用于视频文本检索。但是，另一个挑战在于相似性聚集问题，该问题旨在将细粒度和跨粒度相似性矩阵与实例级别的相似性汇总。为了应对这一挑战，我们提出了对相似性矩阵（AOSM）模块的关注，以使模型重点放在基本帧和单词之间的对比度上，从而降低了不必要的帧和单词对检索结果的影响。 X-CLIP具有多透明的对比度和提议的AOSM模块，在五个广泛使用的视频文本检索数据集上取得了出色的性能，包括MSR-VTT（49.3 R@1），MSVD（50.4 R@1），LSMDC（26.11）（26.1 r@1），didemo（47.8 r@1）和ActivityNet（46.2 r@1）。它的表现优于先前的最先前， +6.3％， +6.6％， +11.1％， +6.7％， +3.8％的相对改善对这些基准测试，这表明了多透明的对比度和AOSM的优势。

This work explores an efficient approach to establish a foundational video-text model for tasks including open-vocabulary video classification, text-to-video retrieval, video captioning and video question-answering. We present VideoCoCa that reuses a pretrained image-text contrastive captioner (CoCa) model and adapt it to video-text tasks with minimal extra training. While previous works adapt image-text models with various cross-frame fusion modules (for example, cross-frame attention layer or perceiver resampler) and finetune the modified architecture on video-text data, we surprisingly find that the generative attentional pooling and contrastive attentional pooling layers in the image-text CoCa design are instantly adaptable to ``flattened frame embeddings'', yielding a strong zero-shot transfer baseline for many video-text tasks. Specifically, the frozen image encoder of a pretrained image-text CoCa takes each video frame as inputs and generates \(N\) token embeddings per frame for totally \(T\) video frames. We flatten \(N \times T\) token embeddings as a long sequence of frozen video representation and apply CoCa's generative attentional pooling and contrastive attentional pooling on top. All model weights including pooling layers are directly loaded from an image-text CoCa pretrained model. Without any video or video-text data, VideoCoCa's zero-shot transfer baseline already achieves state-of-the-art results on zero-shot video classification on Kinetics 400/600/700, UCF101, HMDB51, and Charades, as well as zero-shot text-to-video retrieval on MSR-VTT and ActivityNet Captions. We also explore lightweight finetuning on top of VideoCoCa, and achieve strong results on video question-answering (iVQA, MSRVTT-QA, MSVD-QA) and video captioning (MSR-VTT, ActivityNet, Youcook2). Our approach establishes a simple and effective video-text baseline for future research.

本文介绍了Omnivl，这是一种新的基础模型，旨在使用一种通用体系结构来支持图像语言和视频语言任务。它为图像和视频输入采用了统一的基于变压器的视觉编码器，因此可以执行联合图像语言和视频语言预处理。我们首次证明了这样的范式受益于图像和视频任务，而不是传统的单向传输（例如，使用图像语言来帮助视频语言）。为此，我们提出了对图像语言和视频语言的脱钩关节预处理，以有效地将视觉模型分解为空间和时间维度，并在图像和视频任务上获得性能提升。此外，我们引入了一种新颖的统一视觉对比度（UNIVLC）损失，以利用图像文本，视频文本，图像标签（例如，图像分类），视频标签（例如，视频动作识别）在一起受到监督和吵闹的监督预处理数据都尽可能多地利用。无需额外的任务适配器，Omnivl可以同时支持仅视觉任务（例如，图像分类，视频操作识别），跨模式对齐任务（例如，图像/视频 - 文本检索）和多模式理解和生成任务（例如，图像/视频问答，字幕）。我们在各种下游任务上评估Omnivl，并以相似的模型大小和数据量表获得最新的或竞争结果。

探索大规模预处理的基础模型对计算机视觉具有重大兴趣，因为这些模型可以快速转移到许多下游任务中。本文介绍了对比字幕（COCA），这是一种极简主义的设计，旨在为图像文本编码器编码器基础模型预算与对比度损失和字幕损失，从而从剪辑和诸如simvlm之类的生成方法之类的对比方法中包含模型能力。与所有解码器层都参与编码器输出的标准编码器 - 模块变压器相反，可口可乐省略了解码器层的上半部分的交叉注意，以编码单峰文本表示，并串联到剩余的解码器层，这些解码器与图像编码器相交的解码器层多模式图像文本表示。除了对多模态解码器输出的字幕损失外，我们还应用了单峰图像和文本嵌入之间的对比损失，该输出可以预测文本令牌自动加压。通过共享相同的计算图，可以用最小的开销有效地计算两个培训目标。可口可乐是端到端和从头开始的网络尺度alt-text数据和带注释的图像，通过将所有标签视为文本，无缝地统一自然语言监督以进行表示。从经验上讲，可口可乐通过零拍传输或在广泛的下游任务上进行零摄像转移或最少的特定任务适应，跨越视觉识别（Imagenet，Kinetics-400/600/700，瞬间， ），交叉模式检索（MSCOCO，FLICKR30K，MSR-VTT），多模式理解（VQA，SNLI-VE，NLVR2）和图像字幕（MSCOCO，NOCAPS）。值得注意的是，在Imagenet分类方面，COCA获得了86.3％的TOP-1准确性，带有冷冻编码器和学习的分类头90.6％，以及带有填充编码器的Imagenet上的新最先进的91.0％Top-1 Top-1精度。

Large-scale multi-modal training with image-text pairs imparts strong generalization to CLIP model. Since training on a similar scale for videos is infeasible, recent approaches focus on the effective transfer of image-based CLIP to the video domain. In this pursuit, new parametric modules are added to learn temporal information and inter-frame relationships which require meticulous design efforts. Furthermore, when the resulting models are learned on videos, they tend to overfit on the given task distribution and lack in generalization aspect. This begs the following question: How to effectively transfer image-level CLIP representations to videos? In this work, we show that a simple Video Fine-tuned CLIP (ViFi-CLIP) baseline is generally sufficient to bridge the domain gap from images to videos. Our qualitative analysis illustrates that the frame-level processing from CLIP image-encoder followed by feature pooling and similarity matching with corresponding text embeddings helps in implicitly modeling the temporal cues within ViFi-CLIP. Such fine-tuning helps the model to focus on scene dynamics, moving objects and inter-object relationships. For low-data regimes where full fine-tuning is not viable, we propose a `bridge and prompt' approach that first uses fine-tuning to bridge the domain gap and then learns prompts on language and vision side to adapt CLIP representations. We extensively evaluate this simple yet strong baseline on zero-shot, base-to-novel generalization, few-shot and fully supervised settings across five video benchmarks. Our code is available at https://github.com/muzairkhattak/ViFi-CLIP.

视频文本预训练（VTP）旨在从大规模的网络视频中学习可转移的代表。迄今为止，几乎所有现有的VTP方法都仅限于基于检索的下游任务，例如视频检索，而它们在基于本地化的任务（例如时间基础）上的转移潜力不足。在本文中，我们实验分析并证明了当前VTP方法与本地化任务的不相容性，并提出了一种新颖的面向定位的视频文本预训练框架，称为LocvTP。具体而言，我们执行细粒对比度对准作为通过剪贴字对数发现方案对粗粒粒度的补充。为了进一步增强学习功能的时间推理能力，我们提出了一个上下文投影头和暂时意识的对比损失，以感知上下文关系。对六个数据集的四个下游任务进行的广泛实验表明，我们的LOCVTP在基于检索和基于本地化的任务上都达到了最先进的性能。此外，我们进行了全面的消融研究和彻底的分析，以探索最佳的模型设计和培训策略。

构建一个通用视频语言模型，用于解决各种视频理解任务（例如，文本视频检索，视频问答）是对机器学习领域的开放挑战。为了实现这一目标，最近的尝试训练模型，通常由单峰和跨模式的特征编码器组成，并具有受监督或成对的对比度的预文本任务。尽管提供了有吸引力的通用性，但最终的模型必须在效率和性能之间妥协。我们认为这些缺陷是由它们的预训练策略\ Textemdash引起的，它们不能很好地对齐和融合不同方式的特征。然后，我们将三叶草（一种相关的视频预培训方法）介绍给一个通用的视频语言模型，该模型用于解决既不效率也不妥协的多个视频理解任务。它通过新的三模式比对预训练任务来改善跨模式特征对齐和融合。此外，我们建议通过合并蒙面样品的学习和新颖的成对排名损失来增强三模式对齐。三叶草表现出了出色的一般性。它在多个下游任务上建立了新的最新技术，包括零射击和微调设置的三个检索任务，以及八个视频问答任务。代码和预培训模型将在https://github.com/leeyn-43/clover上发布。

The last several years have witnessed remarkable progress in video-and-language (VidL) understanding. However, most modern VidL approaches use complex and specialized model architectures and sophisticated pretraining protocols, making the reproducibility, analysis and comparisons of these frameworks difficult. Hence, instead of proposing yet another new VidL model, this paper conducts a thorough empirical study demystifying the most important factors in the VidL model design. Among the factors that we investigate are (i) the spatiotemporal architecture design, (ii) the multimodal fusion schemes, (iii) the pretraining objectives, (iv) the choice of pretraining data, (v) pretraining and finetuning protocols, and (vi) dataset and model scaling. Our empirical study reveals that the most important design factors include: temporal modeling, video-to-text multimodal fusion, masked modeling objectives, and joint training on images and videos. Using these empirical insights, we then develop a step-by-step recipe, dubbed VindLU, for effective VidL pretraining. Our final model trained using our recipe achieves comparable or better than state-of-the-art results on several VidL tasks without relying on external CLIP pretraining. In particular, on the text-to-video retrieval task, our approach obtains 61.2% on DiDeMo, and 55.0% on ActivityNet, outperforming current SOTA by 7.8% and 6.1% respectively. Furthermore, our model also obtains state-of-the-art video question-answering results on ActivityNet-QA, MSRVTT-QA, MSRVTT-MC and TVQA. Our code and pretrained models are publicly available at: https://github.com/klauscc/VindLU.

预先训练的图像文本模型（如剪辑）已经证明了从大规模的Web收集的图像文本数据中学到的视觉表示的强大力量。鉴于学习良好的视觉特征，一些现有的作品将图像表示转移到视频域并取得良好的结果。但是，如何利用图像语言预训练的模型（例如，剪辑）进行视频培训（后培训）仍在探索。在本文中，我们研究了两个问题：1）阻碍后期剪辑的因素是什么因素，以进一步提高视频语言任务的性能？ 2）如何减轻这些因素的影响？通过一系列比较实验和分析，我们发现语言源之间的数据量表和域间隙具有很大的影响。由这些动机，我们提出了一种配备了视频代理机制的Omnisource跨模式学习方法，即剪辑，即剪辑VIP。广泛的结果表明，我们的方法可以提高视频检索的剪辑的性能。我们的模型还可以在包括MSR-VTT，DIDEMO，LSMDC和ActivityNet在内的各种数据集上实现SOTA结果。我们在https://github.com/microsoft/xpretrain/tree/main/main/main/clip-vip上发布了代码和预训练的剪辑模型。

在本报告中，我们介绍了图像文本模型的适应性，以进行长期行动预期。我们的视频 +剪辑框架利用了大规模训练的配对图像文本模型：剪辑和视频编码器慢速网络。剪辑嵌入提供了对与操作相关的对象的细粒度理解，而慢速网络负责在几帧的视频片段中对时间信息进行建模。我们表明，从两个编码器获得的功能相互互补，因此在长期行动预期的任务上，在EGO4D上的基线表现优于基线。我们的代码可在github.com/srijandas07/clip_baseline_lta_ego4d上找到。

Video-language pre-training has advanced the performance of various downstream video-language tasks. However, most previous methods directly inherit or adapt typical image-language pre-training paradigms to video-language pre-training, thus not fully exploiting the unique characteristic of video, i.e., temporal. In this paper, we propose a Hierarchical Temporal-Aware video-language pre-training framework, HiTeA, with two novel pre-training tasks for modeling cross-modal alignment between moments and texts as well as the temporal relations of video-text pairs. Specifically, we propose a cross-modal moment exploration task to explore moments in videos, which results in detailed video moment representation. Besides, the inherent temporal relations are captured by aligning video-text pairs as a whole in different time resolutions with multi-modal temporal relation exploration task. Furthermore, we introduce the shuffling test to evaluate the temporal reliance of datasets and video-language pre-training models. We achieve state-of-the-art results on 15 well-established video-language understanding and generation tasks, especially on temporal-oriented datasets (e.g., SSv2-Template and SSv2-Label) with 8.6% and 11.1% improvement respectively. HiTeA also demonstrates strong generalization ability when directly transferred to downstream tasks in a zero-shot manner. Models and demo will be available on ModelScope.

The canonical approach to video-and-language learning (e.g., video question answering) dictates a neural model to learn from offline-extracted dense video features from vision models and text features from language models. These feature extractors are trained independently and usually on tasks different from the target domains, rendering these fixed features sub-optimal for downstream tasks. Moreover, due to the high computational overload of dense video features, it is often difficult (or infeasible) to plug feature extractors directly into existing approaches for easy finetuning. To provide a remedy to this dilemma, we propose a generic framework CLIPBERT that enables affordable endto-end learning for video-and-language tasks, by employing sparse sampling, where only a single or a few sparsely sampled short clips from a video are used at each training step. Experiments on text-to-video retrieval and video question answering on six datasets demonstrate that CLIP-BERT outperforms (or is on par with) existing methods that exploit full-length videos, suggesting that end-to-end learning with just a few sparsely sampled clips is often more accurate than using densely extracted offline features from full-length videos, proving the proverbial less-is-more principle. Videos in the datasets are from considerably different domains and lengths, ranging from 3-second genericdomain GIF videos to 180-second YouTube human activity videos, showing the generalization ability of our approach. Comprehensive ablation studies and thorough analyses are provided to dissect what factors lead to this success. Our code is publicly available. 1 * Equal contribution.

我们提出了MACLR，这是一种新颖的方法，可显式执行从视觉和运动方式中学习的跨模式自我监督的视频表示。与以前的视频表示学习方法相比，主要关注学习运动线索的研究方法是隐含的RGB输入，MACLR丰富了RGB视频片段的标准对比度学习目标，具有运动途径和视觉途径之间的跨模式学习目标。我们表明，使用我们的MACLR方法学到的表示形式更多地关注前景运动区域，因此可以更好地推广到下游任务。为了证明这一点，我们在五个数据集上评估了MACLR，以进行动作识别和动作检测，并在所有数据集上展示最先进的自我监督性能。此外，我们表明MACLR表示可以像在UCF101和HMDB51行动识别的全面监督下所学的表示一样有效，甚至超过了对Vidsitu和SSV2的行动识别的监督表示，以及对AVA的动作检测。

In this paper, we introduce ActBERT for self-supervised learning of joint video-text representations from unlabeled data. First, we leverage global action information to catalyze mutual interactions between linguistic texts and local regional objects. It uncovers global and local visual clues from paired video sequences and text descriptions for detailed visual and text relation modeling. Second, we introduce a TaNgled Transformer block (TNT) to encode three sources of information, i.e., global actions, local regional objects, and linguistic descriptions. Global-local correspondences are discovered via judicious clues extraction from contextual information. It enforces the joint video-text representation to be aware of fine-grained objects as well as global human intention. We validate the generalization capability of ActBERT on downstream video-and-language tasks, i.e., text-video clip retrieval, video captioning, video question answering, action segmentation, and action step localization. ActBERT significantly outperforms the stateof-the-art, demonstrating its superiority in video-text representation learning.actbct * This work was done when Linchao Zhu visited Baidu Research. Yi Yang is the corresponding author.