视频识别是由端到端学习范式主导的 - 首先初始化具有预审预周化图像模型的视频识别模型,然后对视频进行端到端培训。这使视频网络能够受益于验证的图像模型。但是,这需要大量的计算和内存资源,以便在视频上进行填充以及直接使用预审计的图像功能的替代方案,而无需填充图像骨架会导致结果不足。幸运的是,在对比视力语言预训练(剪辑)方面的最新进展为视觉识别任务的新途径铺平了道路。这些模型在大型开放式图像文本对数据上进行了预测,以丰富的语义学习强大的视觉表示。在本文中,我们介绍了有效的视频学习(EVL) - 一种有效的框架,用于直接训练具有冷冻剪辑功能的高质量视频识别模型。具体来说,我们采用轻型变压器解码器并学习查询令牌,从剪辑图像编码器中动态收集帧级空间特征。此外,我们在每个解码器层中采用局部时间模块,以发现相邻帧及其注意力图的时间线索。我们表明,尽管有效地使用冷冻的骨干训练,但我们的模型在各种视频识别数据集上学习了高质量的视频表示。代码可在https://github.com/opengvlab/feld-video-rencognition上找到。
translated by 谷歌翻译
尽管完全监督的人类骨架序列建模成功,但使用自我监督的预训练进行骨架序列表示学习一直是一个活跃的领域,因为很难在大规模上获取特定于任务的骨骼注释。最近的研究重点是使用对比学习学习视频级别的时间和歧视性信息,但忽略了人类骨骼的层次空间时间。与视频级别的这种表面监督不同,我们提出了一种自我监督的分层预训练方案,该方案纳入了基于层次变压器的骨骼骨骼序列编码器(HI-TRS),以明确捕获空间,短期和长期和长期框架,剪辑和视频级别的时间依赖性分别。为了通过HI-TR评估提出的自我监督预训练方案,我们进行了广泛的实验,涵盖了三个基于骨架的下游任务,包括动作识别,动作检测和运动预测。根据监督和半监督评估协议,我们的方法实现了最新的性能。此外,我们证明了我们的模型在训练阶段中学到的先验知识具有强大的下游任务的转移能力。
translated by 谷歌翻译
长期以来,不同的推荐任务通常需要设计特定于任务的架构和培训目标。结果,很难将学习的知识和表示从一个任务转移到另一个任务,从而限制了现有推荐方法的概括能力,例如,几乎无法将顺序推荐模型应用于审核生成方法。为了解决此类问题,考虑到语言几乎可以描述任何内容,语言基础是表示各种问题或任务的有力媒介,我们提出了一种灵活而统一的文本到文本范式,称为“预绘,个性化的提示和预测范式” (P5)为了推荐,该建议在共享框架中统一了各种建议任务。在P5中,将所有数据(例如用户项目交互,用户描述,项目元数据和用户评论)转换为通用格式 - 自然语言序列。来自自然语言的丰富信息有助于P5捕获更深入的语义,以进行个性化和建议。具体而言,P5在预处理过程中以相同的语言建模目标学习不同的任务。因此,它是各种下游建议任务的基础模型,可以轻松地与其他模式集成,并根据提示启用基于指导的建议。 P5将推荐系统从浅层模型到深模型到大型模型,并将彻底改变推荐系统的技术形式,向通用推荐引擎。借助对不同用户的自适应个性化提示,P5能够以零拍或几种方式进行预测,并大大减少了进行广泛微调的必要性。在几个建议基准中,我们进行实验以显示P5的有效性。我们以\ url {https://github.com/jeykigung/p5}发布源代码。
translated by 谷歌翻译
组活动识别(GAR)检测由短视频剪辑中的一组演员执行的活动。任务需要对场景实体的组成理解和它们之间的关系推理。我们通过将视频建模为一系列令牌来致电GAR,该令牌代表视频中的多尺度语义概念。我们提出了Composer,一种基于多尺度变压器的架构,其在每个规模上通过令牌进行关注的推理,并在合成方面学习群组活动。此外,我们只使用缩小场景偏差的关键点模态并提高模型的泛化能力。我们通过群集中间尺度表示来提高作曲家中的多尺度表示,同时在尺度之间保持一致的群集分配。最后,我们使用辅助预测和新型数据增强(例如,演员丢弃)等技术来帮助模型培训。我们展示了挑战排球数据集的模型的实力和可解释性。作曲家通过Keypoint的模型实现新的最先进的94.5%的准确性。作曲家优于依赖RGB信号的最新GAR方法,并对利用多种方式的方法进行比较。我们的代码将可用。
translated by 谷歌翻译
视觉世界自然地展现了一个长尾的开放类分布,这对现代视觉系统带来了巨大挑战。现有方法可以执行类重新平衡策略或直接改进网络模块以解决问题。然而,他们仍然用有限一套预定义标签训练模型,限制了他们的监督信息并限制了他们对新颖实例的可转移性。新途径上的大型对比视觉普瑞宁普雷宁闪光灯的最新进展,可视识别。利用开放词汇监督,预先染色的对比视觉语言模型学习强大的多模式表示,这是对处理数据缺陷和看不见的概念。通过计算视觉和文本输入之间的语义相似性,可视识别被转换为vision语言匹配问题。灵感来自于此,我们提出了民谣,利用了对比尾识别的对比视觉模型。我们首先通过对特定的长尾目标数据集进行对比学习继续预先预留视觉语言骨干。之后,我们冻结了骨干,进一步采用了额外的适配器层,以增强通过重新采样策略构建的平衡训练样本上的尾级课程的表示。已经在三个流行的长尾识别基准测试中进行了广泛的实验。因此,我们简单有效的方法设定了新的最先进的表演,优于具有大边距的竞争基础。代码在https://github.com/gaopengcuhk/ballad发布。
translated by 谷歌翻译
As natural language processing (NLP) for gender bias becomes a significant interdisciplinary topic, the prevalent data-driven techniques such as large-scale language models suffer from data inadequacy and biased corpus, especially for languages with insufficient resources such as Chinese. To this end, we propose a Chinese cOrpus foR Gender bIas Probing and Mitigation CORGI-PM, which contains 32.9k sentences with high-quality labels derived by following an annotation scheme specifically developed for gender bias in the Chinese context. Moreover, we address three challenges for automatic textual gender bias mitigation, which requires the models to detect, classify, and mitigate textual gender bias. We also conduct experiments with state-of-the-art language models to provide baselines. To our best knowledge, CORGI-PM is the first sentence-level Chinese corpus for gender bias probing and mitigation.
translated by 谷歌翻译
Advances in computer vision and machine learning techniques have led to significant development in 2D and 3D human pose estimation from RGB cameras, LiDAR, and radars. However, human pose estimation from images is adversely affected by occlusion and lighting, which are common in many scenarios of interest. Radar and LiDAR technologies, on the other hand, need specialized hardware that is expensive and power-intensive. Furthermore, placing these sensors in non-public areas raises significant privacy concerns. To address these limitations, recent research has explored the use of WiFi antennas (1D sensors) for body segmentation and key-point body detection. This paper further expands on the use of the WiFi signal in combination with deep learning architectures, commonly used in computer vision, to estimate dense human pose correspondence. We developed a deep neural network that maps the phase and amplitude of WiFi signals to UV coordinates within 24 human regions. The results of the study reveal that our model can estimate the dense pose of multiple subjects, with comparable performance to image-based approaches, by utilizing WiFi signals as the only input. This paves the way for low-cost, broadly accessible, and privacy-preserving algorithms for human sensing.
translated by 谷歌翻译
As an important variant of entity alignment (EA), multi-modal entity alignment (MMEA) aims to discover identical entities across different knowledge graphs (KGs) with multiple modalities like images. However, current MMEA algorithms all adopt KG-level modality fusion strategies but ignore modality differences among individual entities, hurting the robustness to potential noise involved in modalities (e.g., unidentifiable images and relations). In this paper we present MEAformer, a multi-modal entity alignment transformer approach for meta modality hybrid, to dynamically predict the mutual correlation coefficients among modalities for instance-level feature fusion. A modal-aware hard entity replay strategy is also proposed for addressing vague entity details. Extensive experimental results show that our model not only achieves SOTA performance on multiple training scenarios including supervised, unsupervised, iterative, and low resource, but also has limited parameters, optimistic speed, and good interpretability. Our code will be available soon.
translated by 谷歌翻译
Long document retrieval aims to fetch query-relevant documents from a large-scale collection, where knowledge distillation has become de facto to improve a retriever by mimicking a heterogeneous yet powerful cross-encoder. However, in contrast to passages or sentences, retrieval on long documents suffers from the scope hypothesis that a long document may cover multiple topics. This maximizes their structure heterogeneity and poses a granular-mismatch issue, leading to an inferior distillation efficacy. In this work, we propose a new learning framework, fine-grained distillation (FGD), for long-document retrievers. While preserving the conventional dense retrieval paradigm, it first produces global-consistent representations crossing different fine granularity and then applies multi-granular aligned distillation merely during training. In experiments, we evaluate our framework on two long-document retrieval benchmarks, which show state-of-the-art performance.
translated by 谷歌翻译
To improve the performance of the dual-encoder retriever, one effective approach is knowledge distillation from the cross-encoder ranker. Existing works construct the candidate passages following the supervised learning setting where a query is paired with a positive passage and a batch of negatives. However, through empirical observation, we find that even the hard negatives from advanced methods are still too trivial for the teacher to distinguish, preventing the teacher from transferring abundant dark knowledge to the student through its soft label. To alleviate this issue, we propose ADAM, a knowledge distillation framework that can better transfer the dark knowledge held in the teacher with Adaptive Dark exAMples. Different from previous works that only rely on one positive and hard negatives as candidate passages, we create dark examples that all have moderate relevance to the query through mixing-up and masking in discrete space. Furthermore, as the quality of knowledge held in different training instances varies as measured by the teacher's confidence score, we propose a self-paced distillation strategy that adaptively concentrates on a subset of high-quality instances to conduct our dark-example-based knowledge distillation to help the student learn better. We conduct experiments on two widely-used benchmarks and verify the effectiveness of our method.
translated by 谷歌翻译