我们解决了视频动作识别的数据增强问题。视频中的标准增强策略是手工设计的，并随机对可能的增强数据点的空间进行采样，而不知道哪个增强点会更好，或者是通过启发式方法会更好。我们建议学习是什么使良好的视频供行动识别，并仅选择高质量的样本进行增强。特别是，我们选择前景和背景视频的视频合成作为数据增强过程，从而导致各种新样本。我们了解了哪对视频要增加，而无需实际综合它们。这降低了可能的增强空间，这具有两个优势：它节省了计算成本并提高了最终训练的分类器的准确性，因为增强对的质量高于平均水平。我们在整个训练环境中介绍了实验结果：几乎没有射击，半监督和完全监督。我们观察到所有这些都对动力学，UCF101，HMDB51的基准进行了一致的改进，并在设置上实现了有限数据的新最新设置。在半监督环境中，我们看到高达8.6％的改善。

机器学习已经急剧提高，在多模式任务中缩小了人类的准确性差距，例如视觉问题答案（VQA）。但是，尽管人类在不确定的时候可以说“我不知道”（即避免回答问题），但这种能力在多模式研究中被大大忽略了，尽管此问题对VQA的使用很重要，而VQA实际上使用了VQA。设置。在这项工作中，我们为可靠的VQA提出了一个问题制定，我们更喜欢弃权，而不是提供错误的答案。我们首先为多种VQA模型提供了弃戒功能，并分析了它们的覆盖范围，回答的问题的一部分和风险，该部分的错误。为此，我们探索了几种弃权方法。我们发现，尽管最佳性能模型在VQA V2数据集上实现了超过71％的准确性，但通过直接使用模型的SoftMax得分介绍了弃权的选项，限制了它们的少于8％的问题，以达到错误的错误风险（即1％）。这促使我们利用多模式选择功能直接估计预测答案的正确性，我们显示的可以将覆盖率增加，例如，在1％风险下，2.4倍从6.8％到16.3％。尽管分析覆盖范围和风险很重要，但这些指标具有权衡，这使得比较VQA模型具有挑战性。为了解决这个问题，我们还建议对VQA的有效可靠性指标，与弃权相比，将不正确的答案的成本更大。 VQA的这种新问题制定，度量和分析为构建有效和可靠的VQA模型提供了基础，这些模型具有自我意识，并且只有当他们不知道答案时才戒除。

在本文中，我们考虑了从长时间的视频到几分钟的长视频进行分类的问题（例如，烹饪不同的食谱，烹饪不同的食谱，进行不同的家庭装修，创建各种形式的艺术和手工艺品）。准确地对这些活动进行分类，不仅需要识别构成任务的单个步骤，还需要捕获其时间依赖性。这个问题与传统的动作分类大不相同，在传统的动作分类中，模型通常在跨越几秒钟的视频上进行了优化，并且手动修剪以包含简单的原子动作。虽然步骤注释可以使模型的培训能够识别程序活动的各个步骤，但由于长时间视频中手动注释时间界的超级注释，因此该领域的现有大规模数据集不包括此类段标签。为了解决这个问题，我们建议通过利用文本知识库（Wikihow）的遥远监督来自动确定教学视频中的步骤，其中包括对执行各种复杂活动所需的步骤的详细描述。我们的方法使用语言模型来匹配视频中自动转录的语音，以在知识库中逐步描述。我们证明，经过训练的视频模型可以识别这些自动标记的步骤（无手动监督）产生了在四个下游任务上实现卓越的概括性能的表示：识别程序活动，步骤分类，步骤预测和以自我为中心的视频分类。

最先进的愿景和愿景和语言模型依靠大规模的Visio-linguisting预借鉴，以获得各种下游任务的良好性能。通常，这种模型通常是跨模态（对比）或多模态（具有早期融合）但不是两者;它们通常只针对特定的方式或任务。有希望的方向将是使用单一整体普遍模型，作为“基础”，目标是一次性的所有方式 - 真正的视觉和语言基础模型应该擅长视力任务，语言任务和交叉和多数模态视觉和语言任务。我们将Flava介绍在这样的模型中，并在跨越这些目标模式的广泛的35个任务上展示令人印象深刻的性能。

零射击动作识别是识别无视觉示例的识别性类别的任务，只有在没有看到看到的类别的seman-tic嵌入方式。问题可以看作是学习一个函数，该函数可以很好地讲述不见的阶级实例，而不会在类之间失去歧视。神经网络可以模拟视觉类别之间的复杂边界，从而将其作为监督模型的成功范围。但是，这些高度专业化的类边界可能不会从看不见的班级转移到看不见的类别。在本文中，我们提出了基于质心的表示，该表示将视觉和语义表示，同时考虑所有训练样本，通过这种方式，对看不见的课程的实例很好。我们使用强化学习对群集进行优化，这对我们的工作方法表明了至关重要的。我们称提出的甲壳类动物的命名为Claster，并观察到它在所有标准数据集中始终超过最先进的方法，包括UCF101，HMDB51和奥运会运动；在Thestandard Zero-shot评估和广义零射击学习中。此外，我们表明我们的模型在图像域也可以进行com的性能，在许多设置中表现出色。

The long-tail distribution of the visual world poses great challenges for deep learning based classification models on how to handle the class imbalance problem. Existing solutions usually involve class-balancing strategies, e.g. by loss re-weighting, data re-sampling, or transfer learning from head-to tail-classes, but most of them adhere to the scheme of jointly learning representations and classifiers. In this work, we decouple the learning procedure into representation learning and classification, and systematically explore how different balancing strategies affect them for long-tailed recognition. The findings are surprising: (1) data imbalance might not be an issue in learning high-quality representations; (2) with representations learned with the simplest instance-balanced (natural) sampling, it is also possible to achieve strong long-tailed recognition ability by adjusting only the classifier. We conduct extensive experiments and set new state-of-the-art performance on common long-tailed benchmarks like ImageNet-LT, Places-LT and iNaturalist, showing that it is possible to outperform carefully designed losses, sampling strategies, even complex modules with memory, by using a straightforward approach that decouples representation and classification. Our code is available at https://github.com/facebookresearch/classifier-balancing.

In lifelong learning, the learner is presented with a sequence of tasks, incrementally building a data-driven prior which may be leveraged to speed up learning of a new task. In this work, we investigate the efficiency of current lifelong approaches, in terms of sample complexity, computational and memory cost. Towards this end, we first introduce a new and a more realistic evaluation protocol, whereby learners observe each example only once and hyper-parameter selection is done on a small and disjoint set of tasks, which is not used for the actual learning experience and evaluation. Second, we introduce a new metric measuring how quickly a learner acquires a new skill. Third, we propose an improved version of GEM (Lopez-Paz & Ranzato, 2017), dubbed Averaged GEM (A-GEM), which enjoys the same or even better performance as GEM, while being almost as computationally and memory efficient as EWC and other regularizationbased methods. Finally, we show that all algorithms including A-GEM can learn even more quickly if they are provided with task descriptors specifying the classification tasks under consideration. Our experiments on several standard lifelong learning benchmarks demonstrate that A-GEM has the best trade-off between accuracy and efficiency. 1

Humans can learn in a continuous manner. Old rarely utilized knowledge can be overwritten by new incoming information while important, frequently used knowledge is prevented from being erased. In artificial learning systems, lifelong learning so far has focused mainly on accumulating knowledge over tasks and overcoming catastrophic forgetting. In this paper, we argue that, given the limited model capacity and the unlimited new information to be learned, knowledge has to be preserved or erased selectively. Inspired by neuroplasticity, we propose a novel approach for lifelong learning, coined Memory Aware Synapses (MAS). It computes the importance of the parameters of a neural network in an unsupervised and online manner. Given a new sample which is fed to the network, MAS accumulates an importance measure for each parameter of the network, based on how sensitive the predicted output function is to a change in this parameter. When learning a new task, changes to important parameters can then be penalized, effectively preventing important knowledge related to previous tasks from being overwritten. Further, we show an interesting connection between a local version of our method and Hebb's rule, which is a model for the learning process in the brain. We test our method on a sequence of object recognition tasks and on the challenging problem of learning an embedding for predicting <subject, predicate, object> triplets. We show state-of-the-art performance and, for the first time, the ability to adapt the importance of the parameters based on unlabeled data towards what the network needs (not) to forget, which may vary depending on test conditions.

Visual question answering is fundamentally compositional in nature-a question like where is the dog? shares substructure with questions like what color is the dog? and where is the cat? This paper seeks to simultaneously exploit the representational capacity of deep networks and the compositional linguistic structure of questions. We describe a procedure for constructing and learning neural module networks, which compose collections of jointly-trained neural "modules" into deep networks for question answering. Our approach decomposes questions into their linguistic substructures, and uses these structures to dynamically instantiate modular networks (with reusable components for recognizing dogs, classifying colors, etc.). The resulting compound networks are jointly trained. We evaluate our approach on two challenging datasets for visual question answering, achieving state-of-the-art results on both the VQA natural image dataset and a new dataset of complex questions about abstract shapes.

We study the ability of foundation models to learn representations for classification that are transferable to new, unseen classes. Recent results in the literature show that representations learned by a single classifier over many classes are competitive on few-shot learning problems with representations learned by special-purpose algorithms designed for such problems. We offer an explanation for this phenomenon based on the concept of class-features variability collapse, which refers to the training dynamics of deep classification networks where the feature embeddings of samples belonging to the same class tend to concentrate around their class means. More specifically, we examine the few-shot error of the learned feature map, which is the classification error of the nearest class-center classifier using centers learned from a small number of random samples from each class. Assuming that the classes appearing in the data are selected independently from a distribution, we show that the few-shot error generalizes from the training data to unseen test data, and we provide an upper bound on the expected few-shot error for new classes (selected from the same distribution) using the average few-shot error for the source classes. Additionally, we show that the few-shot error on the training data can be upper bounded using the degree of class-features variability collapse. This suggests that foundation models can provide feature maps that are transferable to new downstream tasks even with limited data available.