Hopfield networks and Boltzmann machines (BMs) are fundamental energy-based neural network models. Recent studies on modern Hopfield networks have broaden the class of energy functions and led to a unified perspective on general Hopfield networks including an attention module. In this letter, we consider the BM counterparts of modern Hopfield networks using the associated energy functions, and study their salient properties from a trainability perspective. In particular, the energy function corresponding to the attention module naturally introduces a novel BM, which we refer to as attentional BM (AttnBM). We verify that AttnBM has a tractable likelihood function and gradient for a special case and is easy to train. Moreover, we reveal the hidden connections between AttnBM and some single-layer models, namely the Gaussian--Bernoulli restricted BM and denoising autoencoder with softmax units. We also investigate BMs introduced by other energy functions, and in particular, observe that the energy function of dense associative memory models gives BMs belonging to Exponential Family Harmoniums.

Denoising autoencoders have been previously shown to be competitive alternatives to Restricted Boltzmann Machines for unsupervised pre-training of each layer of a deep architecture. We show that a simple denoising autoencoder training criterion is equivalent to matching the score (with respect to the data) of a specific energy based model to that of a non-parametric Parzen density estimator of the data. This yields several useful insights. It defines a proper probabilistic model for the denoising autoencoder technique which makes it in principle possible to sample from them or to rank examples by their energy. It suggests a different way to apply score matching that is related to learning to denoise and does not require computing second derivatives. It justifies the use of tied weights between the encoder and decoder, and suggests ways to extend the success of denoising autoencoders to a larger family of energy-based models.

这是关于Boltzmann机器（BM），受限玻尔兹曼机器（RBM）和Deep信念网络（DBN）的教程和调查论文。我们从概率图形模型，Markov随机字段，Gibbs采样，统计物理学，ISING模型和Hopfield网络的必需背景开始。然后，我们介绍BM和RBM的结构。解释了可见变量和隐藏变量的条件分布，RBM中的GIBBS采样以生成变量，通过最大似然估计训练BM和RBM以及对比度差异。然后，我们讨论变量的不同可能的离散和连续分布。我们介绍有条件的RBM及其训练方式。最后，我们将深度信念网络解释为RBM模型的一堆。本文有关玻尔兹曼机器的论文在包括数据科学，统计，神经计算和统计物理学在内的各个领域都有用。

这项正在进行的工作旨在为统计学习提供统一的介绍，从诸如GMM和HMM等经典模型到现代神经网络（如VAE和扩散模型）缓慢地构建。如今，有许多互联网资源可以孤立地解释这一点或新的机器学习算法，但是它们并没有（也不能在如此简短的空间中）将这些算法彼此连接起来，或者与统计模型的经典文献相连现代算法出现了。同样明显缺乏的是一个单一的符号系统，尽管对那些已经熟悉材料的人（如这些帖子的作者）不满意，但对新手的入境造成了重大障碍。同样，我的目的是将各种模型（尽可能）吸收到一个用于推理和学习的框架上，表明（以及为什么）如何以最小的变化将一个模型更改为另一个模型（其中一些是新颖的，另一些是文献中的）。某些背景当然是必要的。我以为读者熟悉基本的多变量计算，概率和统计以及线性代数。这本书的目标当然不是​​完整性，而是从基本知识到过去十年中极强大的新模型的直线路径或多或少。然后，目标是补充而不是替换，诸如Bishop的\ emph {模式识别和机器学习}之类的综合文本，该文本现在已经15岁了。

This paper presents a tutorial introduction to the use of variational methods for inference and learning in graphical models (Bayesian networks and Markov random fields). We present a number of examples of graphical models, including the QMR-DT database, the sigmoid belief network, the Boltzmann machine, and several variants of hidden Markov models, in which it is infeasible to run exact inference algorithms. We then introduce variational methods, which exploit laws of large numbers to transform the original graphical model into a simplified graphical model in which inference is efficient. Inference in the simpified model provides bounds on probabilities of interest in the original model. We describe a general framework for generating variational transformations based on convex duality. Finally we return to the examples and demonstrate how variational algorithms can be formulated in each case.

我们考虑训练神经网络以存储具有最大噪声稳健性的图案的问题。从最佳权重和状态更新规则方面，解决方案是通过训练每个单独的神经元来执行内核分类或以最小权重规范执行核的分类而得出的。通过将此方法应用于馈送和经常性网络，我们得出了最佳网络，包括特殊情况，包括过去几年中提出的许多异性和自动缔合性记忆模型，例如现代Hopfield Networks和Kanerva's稀疏的分布式内存。我们概括了Kanerva的模型，并展示了一种设计内核内存网络的简单方法，该内核存储网络可以存储具有有限吸引力盆地的连续值数字数量。内核内存网络的框架提供了一种简单而直观的方式来了解以前的存储器模型的存储容量，并允许从树突非线性和突触聚类中进行新的生物学解释。

The success of machine learning algorithms generally depends on data representation, and we hypothesize that this is because different representations can entangle and hide more or less the different explanatory factors of variation behind the data. Although specific domain knowledge can be used to help design representations, learning with generic priors can also be used, and the quest for AI is motivating the design of more powerful representation-learning algorithms implementing such priors. This paper reviews recent work in the area of unsupervised feature learning and deep learning, covering advances in probabilistic models, auto-encoders, manifold learning, and deep networks. This motivates longer-term unanswered questions about the appropriate objectives for learning good representations, for computing representations (i.e., inference), and the geometrical connections between representation learning, density estimation and manifold learning.

指数族在机器学习中广泛使用，包括连续和离散域中的许多分布（例如，通过SoftMax变换，Gaussian，Dirichlet，Poisson和分类分布）。这些家庭中的每个家庭的分布都有固定的支持。相比之下，对于有限域而言，最近在SoftMax稀疏替代方案（例如Sparsemax，$ \ alpha $ -entmax和Fusedmax）的稀疏替代方案中导致了带有不同支持的分布。本文基于几种技术贡献，开发了连续分布的稀疏替代方案：首先，我们定义了$ \ omega $ regultion的预测图和任意域的Fenchel-young损失（可能是无限或连续的）。对于线性参数化的家族，我们表明，Fenchel-Young损失的最小化等效于统计的矩匹配，从而概括了指数家族的基本特性。当$ \ omega $是带有参数$ \ alpha $的Tsallis negentropy时，我们将获得````trabormed rompential指数）''，其中包括$ \ alpha $ -entmax和sparsemax和sparsemax（$ \ alpha = 2 $）。对于二次能量函数，产生的密度为$ \ beta $ -Gaussians，椭圆形分布的实例，其中包含特殊情况，即高斯，双重量级，三人级和epanechnikov密度，我们为差异而得出了差异的封闭式表达式， Tsallis熵和Fenchel-Young损失。当$ \ Omega $是总变化或Sobolev正常化程序时，我们将获得Fusedmax的连续版本。最后，我们引入了连续的注意机制，从\ {1、4/3、3/3、3/2、2 \} $中得出有效的梯度反向传播算法。使用这些算法，我们证明了我们的稀疏连续分布，用于基于注意力的音频分类和视觉问题回答，表明它们允许参加时间间隔和紧凑区域。

Principal Component Analysis (PCA) and its exponential family extensions have three components: observations, latents and parameters of a linear transformation. We consider a generalised setting where the canonical parameters of the exponential family are a nonlinear transformation of the latents. We show explicit relationships between particular neural network architectures and the corresponding statistical models. We find that deep equilibrium models -- a recently introduced class of implicit neural networks -- solve maximum a-posteriori (MAP) estimates for the latents and parameters of the transformation. Our analysis provides a systematic way to relate activation functions, dropout, and layer structure, to statistical assumptions about the observations, thus providing foundational principles for unsupervised DEQs. For hierarchical latents, individual neurons can be interpreted as nodes in a deep graphical model. Our DEQ feature maps are end-to-end differentiable, enabling fine-tuning for downstream tasks.

当我们希望将其用作生成模型时，任何显式的功能表示$ f $都会受到两个主要障碍的阻碍：设计$ f $，以便采样快速，并估计$ z = \ int f $ ^{ - 1} f $集成到1。随着$ f $本身变得复杂，这变得越来越复杂。在本文中，我们表明，当通过让网络代表目标密度的累积分布函数并应用积极的基本定理，可以通过神经网络对一维条件密度进行建模时，可以精确地计算出$ z $。 。我们还得出了一种快速算法，用于通过逆变换方法从产生的表示。通过将这些原理扩展到更高的维度，我们介绍了\ textbf {神经逆变换采样器（NITS）}，这是一个新颖的深度学习框架，用于建模和从一般，多维，紧凑的概率密度。 NIT是一个高度表达性的密度估计器，具有端到端的可不同性，快速采样以及精确且廉价的可能性评估。我们通过将其应用于现实，高维密度估计任务来证明NIT的适用性：基于CIFAR-10数据集对基于可能性的生成模型，以及基于基准数据集的UCI套件的密度估计，nits可以在其中产生令人信服的结果或超越或超越或超越或超越或超越或超越或超越或超越。艺术状态。

The reparameterization trick enables optimizing large scale stochastic computation graphs via gradient descent. The essence of the trick is to refactor each stochastic node into a differentiable function of its parameters and a random variable with fixed distribution. After refactoring, the gradients of the loss propagated by the chain rule through the graph are low variance unbiased estimators of the gradients of the expected loss. While many continuous random variables have such reparameterizations, discrete random variables lack useful reparameterizations due to the discontinuous nature of discrete states. In this work we introduce CONCRETE random variables-CONtinuous relaxations of disCRETE random variables. The Concrete distribution is a new family of distributions with closed form densities and a simple reparameterization. Whenever a discrete stochastic node of a computation graph can be refactored into a one-hot bit representation that is treated continuously, Concrete stochastic nodes can be used with automatic differentiation to produce low-variance biased gradients of objectives (including objectives that depend on the log-probability of latent stochastic nodes) on the corresponding discrete graph. We demonstrate the effectiveness of Concrete relaxations on density estimation and structured prediction tasks using neural networks.

在神经网络的文献中，Hebbian学习传统上是指Hopfield模型及其概括存储原型的程序（即仅经历过一次形成突触矩阵的确定模式）。但是，机器学习中的“学习”一词是指机器从提供的数据集中提取功能的能力（例如，由这些原型的模糊示例制成），以制作自己的不可用原型的代表。在这里，给定一个示例示例，我们定义了一个有监督的学习协议，通过该协议可以通过该协议来推断原型，并检测到正确的控制参数（包括数据集的大小和质量）以描绘系统性能的相图。我们还证明，对于无结构数据集，配备了该监督学习规则的Hopfield模型等同于受限的Boltzmann机器，这表明了最佳且可解释的培训例程。最后，这种方法被推广到结构化的数据集：我们在分析的数据集中突出显示了一个准剥离组织（让人联想到复制对称性 - 对称性），因此，我们为其（部分）分开，为其（部分）删除层引入了一个附加的“复制性隐藏层”，该证明可以将MNIST分类从75％提高到95％，并提供有关深度体系结构的新观点。

这是一门专门针对STEM学生开发的介绍性机器学习课程。我们的目标是为有兴趣的读者提供基础知识，以在自己的项目中使用机器学习，并将自己熟悉术语作为进一步阅读相关文献的基础。在这些讲义中，我们讨论受监督，无监督和强化学习。注释从没有神经网络的机器学习方法的说明开始，例如原理分析，T-SNE，聚类以及线性回归和线性分类器。我们继续介绍基本和先进的神经网络结构，例如密集的进料和常规神经网络，经常性的神经网络，受限的玻尔兹曼机器，（变性）自动编码器，生成的对抗性网络。讨论了潜在空间表示的解释性问题，并使用梦和对抗性攻击的例子。最后一部分致力于加强学习，我们在其中介绍了价值功能和政策学习的基本概念。

Helmholtz机器（HMS）是由两个Sigmoid信念网络（SBN）组成的一类生成模型，分别用作编码器和解码器。这些模型通常是使用称为唤醒 - 睡眠（WS）的两步优化算法对这些模型进行的，并且最近通过改进版本（例如重新恢复的尾流（RWS）和双向Helmholtz Machines（BIHM））进行了改进版本。 SBN中连接的局部性在与概率模型相关的Fisher信息矩阵中诱导稀疏性，并以细粒粒度的块状结构的形式引起。在本文中，我们利用自然梯度利用该特性来有效地训练SBN和HMS。我们提出了一种新颖的算法，称为“自然重新唤醒”（NRWS），该算法与其标准版本的几何适应相对应。以类似的方式，我们还引入了天然双向Helmholtz机器（NBIHM）。与以前的工作不同，我们将展示如何有效地计算自然梯度，而无需引入Fisher信息矩阵结构的任何近似值。在文献中进行的标准数据集进行的实验表明，NRW和NBIHM不仅在其非几何基准方面，而且在HMS的最先进培训算法方面都具有一致的改善。在训练后，汇聚速度以及对数可能达到的对数似然的值量化了改进。

Perturb-and-MAP offers an elegant approach to approximately sample from an energy-based model (EBM) by computing the maximum-a-posteriori (MAP) configuration of a perturbed version of the model. Sampling in turn enables learning. However, this line of research has been hindered by the general intractability of the MAP computation. Very few works venture outside tractable models, and when they do, they use linear programming approaches, which as we show, have several limitations. In this work, we present perturb-and-max-product (PMP), a parallel and scalable mechanism for sampling and learning in discrete EBMs. Models can be arbitrary as long as they are built using tractable factors. We show that (a) for Ising models, PMP is orders of magnitude faster than Gibbs and Gibbs-with-Gradients (GWG) at learning and generating samples of similar or better quality; (b) PMP is able to learn and sample from RBMs; (c) in a large, entangled graphical model in which Gibbs and GWG fail to mix, PMP succeeds.Preprint. Under review.

Normalizing flows provide a general mechanism for defining expressive probability distributions, only requiring the specification of a (usually simple) base distribution and a series of bijective transformations. There has been much recent work on normalizing flows, ranging from improving their expressive power to expanding their application. We believe the field has now matured and is in need of a unified perspective. In this review, we attempt to provide such a perspective by describing flows through the lens of probabilistic modeling and inference. We place special emphasis on the fundamental principles of flow design, and discuss foundational topics such as expressive power and computational trade-offs. We also broaden the conceptual framing of flows by relating them to more general probability transformations. Lastly, we summarize the use of flows for tasks such as generative modeling, approximate inference, and supervised learning.

期望 - 最大化（EM）算法是一种简单的元叠加，当观察到的数据中缺少测量值或数据由可观察到的数据组成时，它已多年来用作统计推断的方法。它的一般属性进行了充分的研究，而且还有无数方法将其应用于个人问题。在本文中，我们介绍了$ em $ $ and算法，EM算法的信息几何公式及其扩展和应用程序以及各种问题。具体而言，我们将看到，可以制定一个异常稳定推理算法，用于计算通道容量的算法，概率单纯性的参数估计方法，特定的多变量分析方法，例如概率模型中的主要组件分析和模态回归中的主成分分析，基质分解和学习生成模型，这些模型最近从几何学角度引起了深度学习的关注。

无监督的机器学习的目的是删除复杂的高维数据的表示形式，从而解释数据中的重要潜在因素以及操纵它们以生成具有理想功能的新数据。这些方法通常依赖于对抗方案，在该方案中，对代表进行调整以避免歧视者能够重建特定的数据信息（标签）。我们提出了一种简单，有效的方法，即在无需培训对抗歧视器的情况下解开表示形式，并将我们的方法应用于受限的玻尔兹曼机器（RBM），这是最简单的基于代表的生成模型之一。我们的方法依赖于在训练过程中引入对权重的足够约束，这使我们能够将有关标签的信息集中在一小部分潜在变量上。该方法的有效性在MNIST数据集，二维ISING模型和蛋白质家族的分类法上说明了。此外，我们还展示了我们的框架如何从数据的对数模型中计算成本，与其表示形式的删除相关。

文献中已经提出了许多关联记忆的神经网络模型。其中包括经典的Hopfield网络（HNS），稀疏分布式记忆（SDM）以及最近的现代连续Hopfield网络（MCHN），该网络在机器学习中具有与自我注意力的紧密联系。在本文中，我们提出了一个通用框架，以理解此类内存网络的操作，例如三个操作的顺序：相似性，分离和投影。我们将所有这些记忆模型作为我们的一般框架的实例，具有不同的相似性和分离函数。我们将Krotov等人（2020）的数学框架扩展到使用神经元之间仅具有二阶相互作用的神经网络动力学来表达通用的关联存储模型，并得出了一种通用能量函数，该函数是动力学的lyapunov函数。最后，使用我们的框架，我们从经验上研究了这些关联记忆模型使用不同相似性函数的能力，超出了点产品相似性度量，并从经验上证明了欧几里得或曼哈顿距离距离相似性指标在实践中在许多任务中表现出色，从而启用了一项启用一项效果比现有模型更强大的检索和更高的内存能力。

We explore an original strategy for building deep networks, based on stacking layers of denoising autoencoders which are trained locally to denoise corrupted versions of their inputs. The resulting algorithm is a straightforward variation on the stacking of ordinary autoencoders. It is however shown on a benchmark of classification problems to yield significantly lower classification error, thus bridging the performance gap with deep belief networks (DBN), and in several cases surpassing it. Higher level representations learnt in this purely unsupervised fashion also help boost the performance of subsequent SVM classifiers. Qualitative experiments show that, contrary to ordinary autoencoders, denoising autoencoders are able to learn Gabor-like edge detectors from natural image patches and larger stroke detectors from digit images. This work clearly establishes the value of using a denoising criterion as a tractable unsupervised objective to guide the learning of useful higher level representations.