深度神经网络的学习算法通常基于有误后传播(BackProp)的监督端到端随机梯度下降(SGD)培训。 Backprop算法需要大量标记的训练样本才能获得高性能。但是,在许多现实的应用中,即使有很多图像样本,很少有标签被标记,并且必须使用半监督的样品培训策略。 Hebbian学习代表了一种可能采取样本培训的方法;但是,在当前解决方案中,它不能很好地扩展到大型数据集。在本文中,我们提出了FastheBB,这是HEBBIAN学习的有效且可扩展的解决方案,通过1)合并在一批输入上更新计算和聚集,以及2)利用有效的GPU上的有效矩阵乘法算法。在半监督的学习方案中,我们在不同的计算机视觉基准测试方面验证了我们的方法。 FastheBB在训练速度方面最多优于先前的解决方案,尤其是,我们首次能够将HEBBIAN算法带入ImageNet量表。
translated by 谷歌翻译
This work tackles the problem of semi-supervised learning of image classifiers. Our main insight is that the field of semi-supervised learning can benefit from the quickly advancing field of self-supervised visual representation learning. Unifying these two approaches, we propose the framework of self-supervised semi-supervised learning (S 4 L) and use it to derive two novel semi-supervised image classification methods. We demonstrate the effectiveness of these methods in comparison to both carefully tuned baselines, and existing semi-supervised learning methods. We then show that S 4 L and existing semi-supervised methods can be jointly trained, yielding a new state-of-the-art result on semi-supervised ILSVRC-2012 with 10% of labels.
translated by 谷歌翻译
Jitendra Malik once said, "Supervision is the opium of the AI researcher". Most deep learning techniques heavily rely on extreme amounts of human labels to work effectively. In today's world, the rate of data creation greatly surpasses the rate of data annotation. Full reliance on human annotations is just a temporary means to solve current closed problems in AI. In reality, only a tiny fraction of data is annotated. Annotation Efficient Learning (AEL) is a study of algorithms to train models effectively with fewer annotations. To thrive in AEL environments, we need deep learning techniques that rely less on manual annotations (e.g., image, bounding-box, and per-pixel labels), but learn useful information from unlabeled data. In this thesis, we explore five different techniques for handling AEL.
translated by 谷歌翻译
最近,稀疏的培训方法已开始作为事实上的人工神经网络的培训和推理效率的方法。然而,这种效率只是理论上。在实践中,每个人都使用二进制掩码来模拟稀疏性,因为典型的深度学习软件和硬件已针对密集的矩阵操作进行了优化。在本文中,我们采用正交方法,我们表明我们可以训练真正稀疏的神经网络以收获其全部潜力。为了实现这一目标,我们介绍了三个新颖的贡献,这些贡献是专门为稀疏神经网络设计的:(1)平行训练算法及其相应的稀疏实现,(2)具有不可训练的参数的激活功能,以支持梯度流动,以支持梯度流量, (3)隐藏的神经元对消除冗余的重要性指标。总而言之,我们能够打破记录并训练有史以来最大的神经网络在代表力方面训练 - 达到蝙蝠大脑的大小。结果表明,我们的方法具有最先进的表现,同时为环保人工智能时代开辟了道路。
translated by 谷歌翻译
Inspired by progress in unsupervised representation learning for natural language, we examine whether similar models can learn useful representations for images. We train a sequence Transformer to auto-regressively predict pixels, without incorporating knowledge of the 2D input structure. Despite training on low-resolution ImageNet without labels, we find that a GPT-2 scale model learns strong image representations as measured by linear probing, fine-tuning, and low-data classification. On CIFAR-10, we achieve 96.3% accuracy with a linear probe, outperforming a supervised Wide ResNet, and 99.0% accuracy with full fine-tuning, matching the top supervised pretrained models. We are also competitive with self-supervised benchmarks on ImageNet when substituting pixels for a VQVAE encoding, achieving 69.0% top-1 accuracy on a linear probe of our features.
translated by 谷歌翻译
Human observers can learn to recognize new categories of images from a handful of examples, yet doing so with artificial ones remains an open challenge. We hypothesize that data-efficient recognition is enabled by representations which make the variability in natural signals more predictable. We therefore revisit and improve Contrastive Predictive Coding, an unsupervised objective for learning such representations. This new implementation produces features which support state-of-theart linear classification accuracy on the ImageNet dataset. When used as input for non-linear classification with deep neural networks, this representation allows us to use 2-5× less labels than classifiers trained directly on image pixels. Finally, this unsupervised representation substantially improves transfer learning to object detection on the PASCAL VOC dataset, surpassing fully supervised pre-trained ImageNet classifiers.
translated by 谷歌翻译
Continual Learning (CL) is a field dedicated to devise algorithms able to achieve lifelong learning. Overcoming the knowledge disruption of previously acquired concepts, a drawback affecting deep learning models and that goes by the name of catastrophic forgetting, is a hard challenge. Currently, deep learning methods can attain impressive results when the data modeled does not undergo a considerable distributional shift in subsequent learning sessions, but whenever we expose such systems to this incremental setting, performance drop very quickly. Overcoming this limitation is fundamental as it would allow us to build truly intelligent systems showing stability and plasticity. Secondly, it would allow us to overcome the onerous limitation of retraining these architectures from scratch with the new updated data. In this thesis, we tackle the problem from multiple directions. In a first study, we show that in rehearsal-based techniques (systems that use memory buffer), the quantity of data stored in the rehearsal buffer is a more important factor over the quality of the data. Secondly, we propose one of the early works of incremental learning on ViTs architectures, comparing functional, weight and attention regularization approaches and propose effective novel a novel asymmetric loss. At the end we conclude with a study on pretraining and how it affects the performance in Continual Learning, raising some questions about the effective progression of the field. We then conclude with some future directions and closing remarks.
translated by 谷歌翻译
Image classification with small datasets has been an active research area in the recent past. However, as research in this scope is still in its infancy, two key ingredients are missing for ensuring reliable and truthful progress: a systematic and extensive overview of the state of the art, and a common benchmark to allow for objective comparisons between published methods. This article addresses both issues. First, we systematically organize and connect past studies to consolidate a community that is currently fragmented and scattered. Second, we propose a common benchmark that allows for an objective comparison of approaches. It consists of five datasets spanning various domains (e.g., natural images, medical imagery, satellite data) and data types (RGB, grayscale, multispectral). We use this benchmark to re-evaluate the standard cross-entropy baseline and ten existing methods published between 2017 and 2021 at renowned venues. Surprisingly, we find that thorough hyper-parameter tuning on held-out validation data results in a highly competitive baseline and highlights a stunted growth of performance over the years. Indeed, only a single specialized method dating back to 2019 clearly wins our benchmark and outperforms the baseline classifier.
translated by 谷歌翻译
最近对反向传播的近似(BP)减轻了BP的许多计算效率低下和与生物学的不兼容性,但仍然存在重要的局限性。此外,近似值显着降低了基准的准确性,这表明完全不同的方法可能更富有成果。在这里,基于在软冠军全网络中Hebbian学习的最新理论基础上,我们介绍了多层softhebb,即一种训练深神经网络的算法,没有任何反馈,目标或错误信号。结果,它通过避免重量传输,非本地可塑性,层更新的时间锁定,迭代平衡以及(自我)监督或其他反馈信号来实现效率,这在其他方法中是必不可少的。与最先进的生物学知识学习相比,它提高的效率和生物兼容性不能取得准确性的折衷,而是改善了准确性。 MNIST,CIFAR-10,STL-10和IMAGENET上最多五个隐藏层和添加的线性分类器,分别达到99.4%,80.3%,76.2%和27.3%。总之,SOFTHEBB显示出与BP的截然不同的方法,即对几层的深度学习在大脑中可能是合理的,并提高了生物学上的机器学习的准确性。
translated by 谷歌翻译
Neural net classifiers trained on data with annotated class labels can also capture apparent visual similarity among categories without being directed to do so. We study whether this observation can be extended beyond the conventional domain of supervised learning: Can we learn a good feature representation that captures apparent similarity among instances, instead of classes, by merely asking the feature to be discriminative of individual instances?We formulate this intuition as a non-parametric classification problem at the instance-level, and use noisecontrastive estimation to tackle the computational challenges imposed by the large number of instance classes.Our experimental results demonstrate that, under unsupervised learning settings, our method surpasses the stateof-the-art on ImageNet classification by a large margin. Our method is also remarkable for consistently improving test performance with more training data and better network architectures. By fine-tuning the learned feature, we further obtain competitive results for semi-supervised learning and object detection tasks. Our non-parametric model is highly compact: With 128 features per image, our method requires only 600MB storage for a million images, enabling fast nearest neighbour retrieval at the run time.
translated by 谷歌翻译
大量的数据和创新算法使数据驱动的建模成为现代行业的流行技术。在各种数据驱动方法中,潜在变量模型(LVM)及其对应物占主要份额,并在许多工业建模领域中起着至关重要的作用。 LVM通常可以分为基于统计学习的经典LVM和基于神经网络的深层LVM(DLVM)。我们首先讨论经典LVM的定义,理论和应用,该定义和应用既是综合教程,又是对经典LVM的简短申请调查。然后,我们对当前主流DLVM进行了彻底的介绍,重点是其理论和模型体系结构,此后不久就提供了有关DLVM的工业应用的详细调查。上述两种类型的LVM具有明显的优势和缺点。具体而言,经典的LVM具有简洁的原理和良好的解释性,但是它们的模型能力无法解决复杂的任务。基于神经网络的DLVM具有足够的模型能力,可以在复杂的场景中实现令人满意的性能,但它以模型的解释性和效率为例。旨在结合美德并减轻这两种类型的LVM的缺点,并探索非神经网络的举止以建立深层模型,我们提出了一个新颖的概念,称为“轻量级Deep LVM(LDLVM)”。在提出了这个新想法之后,该文章首先阐述了LDLVM的动机和内涵,然后提供了两个新颖的LDLVM,并详尽地描述了其原理,建筑和优点。最后,讨论了前景和机会,包括重要的开放问题和可能的研究方向。
translated by 谷歌翻译
Transfer of pre-trained representations improves sample efficiency and simplifies hyperparameter tuning when training deep neural networks for vision. We revisit the paradigm of pre-training on large supervised datasets and fine-tuning the model on a target task. We scale up pre-training, and propose a simple recipe that we call Big Transfer (BiT). By combining a few carefully selected components, and transferring using a simple heuristic, we achieve strong performance on over 20 datasets. BiT performs well across a surprisingly wide range of data regimes -from 1 example per class to 1 M total examples. BiT achieves 87.5% top-1 accuracy on ILSVRC-2012, 99.4% on CIFAR-10, and 76.3% on the 19 task Visual Task Adaptation Benchmark (VTAB). On small datasets, BiT attains 76.8% on ILSVRC-2012 with 10 examples per class, and 97.0% on CIFAR-10 with 10 examples per class. We conduct detailed analysis of the main components that lead to high transfer performance.
translated by 谷歌翻译
Despite significant advances, the performance of state-of-the-art continual learning approaches hinges on the unrealistic scenario of fully labeled data. In this paper, we tackle this challenge and propose an approach for continual semi-supervised learning -- a setting where not all the data samples are labeled. An underlying issue in this scenario is the model forgetting representations of unlabeled data and overfitting the labeled ones. We leverage the power of nearest-neighbor classifiers to non-linearly partition the feature space and learn a strong representation for the current task, as well as distill relevant information from previous tasks. We perform a thorough experimental evaluation and show that our method outperforms all the existing approaches by large margins, setting a strong state of the art on the continual semi-supervised learning paradigm. For example, on CIFAR100 we surpass several others even when using at least 30 times less supervision (0.8% vs. 25% of annotations).
translated by 谷歌翻译
We introduce a method to train Quantized Neural Networks (QNNs) -neural networks with extremely low precision (e.g., 1-bit) weights and activations, at run-time. At traintime the quantized weights and activations are used for computing the parameter gradients. During the forward pass, QNNs drastically reduce memory size and accesses, and replace most arithmetic operations with bit-wise operations. As a result, power consumption is expected to be drastically reduced. We trained QNNs over the MNIST, CIFAR-10, SVHN and ImageNet datasets. The resulting QNNs achieve prediction accuracy comparable to their 32-bit counterparts. For example, our quantized version of AlexNet with 1-bit weights and 2-bit activations achieves 51% top-1 accuracy. Moreover, we quantize the parameter gradients to 6-bits as well which enables gradients computation using only bit-wise operation. Quantized recurrent neural networks were tested over the Penn Treebank dataset, and achieved comparable accuracy as their 32-bit counterparts using only 4-bits. Last but not least, we programmed a binary matrix multiplication GPU kernel with which it is possible to run our MNIST QNN 7 times faster than with an unoptimized GPU kernel, without suffering any loss in classification accuracy. The QNN code is available online.
translated by 谷歌翻译
深度学习使用由其重量进行参数化的神经网络。通常通过调谐重量来直接最小化给定损耗功能来训练神经网络。在本文中,我们建议将权重重新参数转化为网络中各个节点的触发强度的目标。给定一组目标,可以计算使得发射强度最佳地满足这些目标的权重。有人认为,通过我们称之为级联解压缩的过程,使用培训的目标解决爆炸梯度的问题,并使损失功能表面更加光滑,因此导致更容易,培训更快,以及潜在的概括,神经网络。它还允许更容易地学习更深层次和经常性的网络结构。目标对重量的必要转换有额外的计算费用,这是在许多情况下可管理的。在目标空间中学习可以与现有的神经网络优化器相结合,以额外收益。实验结果表明了使用目标空间的速度,以及改进的泛化的示例,用于全连接的网络和卷积网络,以及调用和处理长时间序列的能力,并使用经常性网络进行自然语言处理。
translated by 谷歌翻译
我们为半监督学习设置提供了一种新的数据增强技术,该技术强调从功能空间最具挑战性的地区学习。从完全监督的参考模型开始,我们首先确定较低的置信度预测。然后,这些样品用于训练变异自动编码器(VAE),该变量可以生成具有相似分布的无限额外图像。最后,使用最初标记的数据和合成生成的标记和未标记的数据,我们以半监视的方式重新训练了一个新模型。我们对两个基准RGB数据集进行实验:CIFAR-100和STL-10,并表明所提出的方案在准确性和鲁棒性方面提高了分类性能,同时就现有的完全监督的方法而产生可比或优越的结果。
translated by 谷歌翻译
监测原位浮游生物的种群对于保留水生生态系统至关重要。浮游生物微生物实际上易受较小的环境扰动的影响,可以反映出随之而来的形态学和动力学修饰。如今,高级自动或半自动采集系统的可用性已允许生产越来越多的浮游生物图像数据。由于大量获得的数据和浮游生物的数字,因此,采用机器学习算法来对此类数据进行分类。为了应对这些挑战,我们提出了有效的无监督学习管道,以提供浮游生物微生物的准确分类。我们构建一组图像描述符,利用两步过程。首先,对预先训练的神经网络提取的功能进行了跨自动编码器(VAE)的培训。然后,我们将学习的潜在空间用作聚类的图像描述符。我们将方法与最新的无监督方法进行了比较,其中一组预定义的手工特征用于浮游生物图像的聚类。所提出的管道优于我们分析中包含的所有浮游生物数据集的基准算法,提供了更好的图像嵌入属性。
translated by 谷歌翻译
我们提出了Parse,这是一种新颖的半监督结构,用于学习强大的脑电图表现以进行情感识别。为了减少大量未标记数据与标记数据有限的潜在分布不匹配,Parse使用成对表示对准。首先,我们的模型执行数据增强,然后标签猜测大量原始和增强的未标记数据。然后将其锐化的标签和标记数据的凸组合锐化。最后,进行表示对准和情感分类。为了严格测试我们的模型,我们将解析与我们实施并适应脑电图学习的几种最先进的半监督方法进行了比较。我们对四个基于公共EEG的情绪识别数据集,种子,种子IV,种子V和Amigos(价和唤醒)进行这些实验。该实验表明,我们提出的框架在种子,种子-IV和Amigos(Valence)中的标记样品有限的情况下,取得了总体最佳效果,同时接近种子V和Amigos中的总体最佳结果(达到第二好) (唤醒)。分析表明,我们的成对表示对齐方式通过减少未标记数据和标记数据之间的分布比对来大大提高性能,尤其是当每类仅1个样本被标记时。
translated by 谷歌翻译
在深度学习研究中,自学学习(SSL)引起了极大的关注,引起了计算机视觉和遥感社区的兴趣。尽管计算机视觉取得了很大的成功,但SSL在地球观测领域的大部分潜力仍然锁定。在本文中,我们对在遥感的背景下为计算机视觉的SSL概念和最新发展提供了介绍,并回顾了SSL中的概念和最新发展。此外,我们在流行的遥感数据集上提供了现代SSL算法的初步基准,从而验证了SSL在遥感中的潜力,并提供了有关数据增强的扩展研究。最后,我们确定了SSL未来研究的有希望的方向的地球观察(SSL4EO),以铺平了两个领域的富有成效的相互作用。
translated by 谷歌翻译
Effective convolutional neural networks are trained on large sets of labeled data. However, creating large labeled datasets is a very costly and time-consuming task. Semi-supervised learning uses unlabeled data to train a model with higher accuracy when there is a limited set of labeled data available. In this paper, we consider the problem of semi-supervised learning with convolutional neural networks. Techniques such as randomized data augmentation, dropout and random max-pooling provide better generalization and stability for classifiers that are trained using gradient descent. Multiple passes of an individual sample through the network might lead to different predictions due to the non-deterministic behavior of these techniques. We propose an unsupervised loss function that takes advantage of the stochastic nature of these methods and minimizes the difference between the predictions of multiple passes of a training sample through the network. We evaluate the proposed method on several benchmark datasets.
translated by 谷歌翻译