Iterative regularization is a classic idea in regularization theory, that has recently become popular in machine learning. On the one hand, it allows to design efficient algorithms controlling at the same time numerical and statistical accuracy. On the other hand it allows to shed light on the learning curves observed while training neural networks. In this paper, we focus on iterative regularization in the context of classification. After contrasting this setting with that of regression and inverse problems, we develop an iterative regularization approach based on the use of the hinge loss function. More precisely we consider a diagonal approach for a family of algorithms for which we prove convergence as well as rates of convergence. Our approach compares favorably with other alternatives, as confirmed also in numerical simulations.
translated by 谷歌翻译
我们介绍和分析结构化的随机零订单下降(S-SZD),这是一种有限的差异方法,该方法在一组$ l \ leq d $正交方向上近似于随机梯度,其中$ d $是环境空间的维度。这些方向是随机选择的,并且可能在每个步骤中发生变化。对于平滑的凸功能,我们几乎可以确保迭代的收敛性和对$ o(d/l k^{ - c})$的功能值的收敛速率,每$ c <1/2 $,这是任意关闭的就迭代次数而言,是随机梯度下降(SGD)。我们的界限还显示了使用$ l $多个方向而不是一个方向的好处。对于满足polyak-{\ l} ojasiewicz条件的非convex函数,我们在这种假设下建立了随机Zeroth Order Order Order算法的第一个收敛速率。我们在数值模拟中证实了我们的理论发现,在数值模拟中,满足假设以及对超参数优化的现实世界问题,观察到S-SZD具有很好的实践性能。
translated by 谷歌翻译
高斯流程优化是一类成功的算法(例如GP-UCB),以通过顺序评估优化黑盒功能。然而,对于具有连续域的功能,高斯过程优化必须依赖于空间的固定离散化,或者在每个评估中解决非凸优化子问题的解决方案。第一种方法可能会对性能产生负面影响,而第二种方法则需要沉重的计算负担。第三种选项最近理论上学习,是自适应地离散功能域。尽管这种方法避免了额外的非凸优化成本,但整体计算复杂性仍然令人望而却步。诸如GP-UCB的算法具有$ O(t ^ 4)$的运行时间,其中$ t $是迭代的数量。在本文中,我们介绍了ADA-BKB(自适应预算的核化强盗),是一种无遗憾的高斯过程优化算法,用于连续域上的功能,可在$ O(t ^ 2 d_ \ text {eff} ^ 2)$ ,$ d_ \ text {eff} $是探索空间的有效维度,其通常小于$ t $。我们将我们的理论调查结果与合成非凸函数的实验以及超参数优化的真实问题进行了证实,确认了所提出的方法的良好实际表现。
translated by 谷歌翻译
Computational units in artificial neural networks follow a simplified model of biological neurons. In the biological model, the output signal of a neuron runs down the axon, splits following the many branches at its end, and passes identically to all the downward neurons of the network. Each of the downward neurons will use their copy of this signal as one of many inputs dendrites, integrate them all and fire an output, if above some threshold. In the artificial neural network, this translates to the fact that the nonlinear filtering of the signal is performed in the upward neuron, meaning that in practice the same activation is shared between all the downward neurons that use that signal as their input. Dendrites thus play a passive role. We propose a slightly more complex model for the biological neuron, where dendrites play an active role: the activation in the output of the upward neuron becomes optional, and instead the signals going through each dendrite undergo independent nonlinear filterings, before the linear combination. We implement this new model into a ReLU computational unit and discuss its biological plausibility. We compare this new computational unit with the standard one and describe it from a geometrical point of view. We provide a Keras implementation of this unit into fully connected and convolutional layers and estimate their FLOPs and weights change. We then use these layers in ResNet architectures on CIFAR-10, CIFAR-100, Imagenette, and Imagewoof, obtaining performance improvements over standard ResNets up to 1.73%. Finally, we prove a universal representation theorem for continuous functions on compact sets and show that this new unit has more representational power than its standard counterpart.
translated by 谷歌翻译
Recent advances in language modeling have enabled new conversational systems. In particular, it is often desirable for people to make choices among specified options when using such systems. We address the problem of reference resolution, when people use natural expressions to choose between real world entities. For example, given the choice `Should we make a Simnel cake or a Pandan cake?' a natural response from a non-expert may be indirect: `let's make the green one'. Reference resolution has been little studied with natural expressions, thus robustly understanding such language has large potential for improving naturalness in dialog, recommendation, and search systems. We create AltEntities (Alternative Entities), a new public dataset of entity pairs and utterances, and develop models for the disambiguation problem. Consisting of 42K indirect referring expressions across three domains, it enables for the first time the study of how large language models can be adapted to this task. We find they achieve 82%-87% accuracy in realistic settings, which while reasonable also invites further advances.
translated by 谷歌翻译
Modern machine learning pipelines are limited due to data availability, storage quotas, privacy regulations, and expensive annotation processes. These constraints make it difficult or impossible to maintain a large-scale model trained on growing annotation sets. Continual learning directly approaches this problem, with the ultimate goal of devising methods where a neural network effectively learns relevant patterns for new (unseen) classes without significantly altering its performance on previously learned ones. In this paper, we address the problem of continual learning for video data. We introduce PIVOT, a novel method that leverages the extensive knowledge in pre-trained models from the image domain, thereby reducing the number of trainable parameters and the associated forgetting. Unlike previous methods, ours is the first approach that effectively uses prompting mechanisms for continual learning without any in-domain pre-training. Our experiments show that PIVOT improves state-of-the-art methods by a significant 27% on the 20-task ActivityNet setup.
translated by 谷歌翻译
In this work, we apply a kinetic version of a bounded confidence consensus model to biomedical segmentation problems. In the presented approach, time-dependent information on the microscopic state of each particle/pixel includes its space position and a feature representing a static characteristic of the system, i.e. the gray level of each pixel. From the introduced microscopic model we derive a kinetic formulation of the model. The large time behavior of the system is then computed with the aid of a surrogate Fokker-Planck approach that can be obtained in the quasi-invariant scaling. We exploit the computational efficiency of direct simulation Monte Carlo methods for the obtained Boltzmann-type description of the problem for parameter identification tasks. Based on a suitable loss function measuring the distance between the ground truth segmentation mask and the evaluated mask, we minimize the introduced segmentation metric for a relevant set of 2D gray-scale images. Applications to biomedical segmentation concentrate on different imaging research contexts.
translated by 谷歌翻译
Chain event graphs are a family of probabilistic graphical models that generalise Bayesian networks and have been successfully applied to a wide range of domains. Unlike Bayesian networks, these models can encode context-specific conditional independencies as well as asymmetric developments within the evolution of a process. More recently, new model classes belonging to the chain event graph family have been developed for modelling time-to-event data to study the temporal dynamics of a process. However, existing model selection algorithms for chain event graphs and its variants rely on all parameters having conjugate priors. This is unrealistic for many real-world applications. In this paper, we propose a mixture modelling approach to model selection in chain event graphs that does not rely on conjugacy. Moreover, we also show that this methodology is more amenable to being robustly scaled than the existing model selection algorithms used for this family. We demonstrate our techniques on simulated datasets.
translated by 谷歌翻译
The detection of state-sponsored trolls acting in information operations is an unsolved and critical challenge for the research community, with repercussions that go beyond the online realm. In this paper, we propose a novel AI-based solution for the detection of state-sponsored troll accounts, which consists of two steps. The first step aims at classifying trajectories of accounts' online activities as belonging to either a state-sponsored troll or to an organic user account. In the second step, we exploit the classified trajectories to compute a metric, namely "troll score", which allows us to quantify the extent to which an account behaves like a state-sponsored troll. As a study case, we consider the troll accounts involved in the Russian interference campaign during the 2016 US Presidential election, identified as Russian trolls by the US Congress. Experimental results show that our approach identifies accounts' trajectories with an AUC close to 99\% and, accordingly, classify Russian trolls and organic users with an AUC of 97\%. Finally, we evaluate whether the proposed solution can be generalized to different contexts (e.g., discussions about Covid-19) and generic misbehaving users, showing promising results that will be further expanded in our future endeavors.
translated by 谷歌翻译
我们介绍并讨论了一个运行时体系结构,该架构将感官数据和分类器与基于逻辑的决策系统集成在一起,并在电子健康系统的背景下,用于康复神经运动障碍儿童。在此应用程序中,儿童以游戏的形式执行康复任务。该系统的主要目的是从可用的传感器和分类器(例如,眼镜跟踪器,运动传感器,情感识别技术)中得出一组儿童当前的认知和行为表现(例如参与,注意力,任务准确性)的参数。 )并做出相应的决定。这些决策通常旨在通过在注意力较低时触发适当的重新参与刺激,改变游戏或使孩子对任务失去兴趣时的困难来改善孩子的表现,因为它太容易了。除了对情绪识别和头部姿势估计的最新技术外,我们还使用了事件计算的概率和认知逻辑编程方言的运行时变体,称为认识论概率概率事件。特别是,该符号框架的概率组成部分允许与机器学习技术的自然接口。我们概述了体系结构及其组件,并通过讨论运行的示例和实验来展示其一些特征。正在考虑逻辑编程理论和实践(TPLP)的出版物。
translated by 谷歌翻译