Nonconvex optimization is central in solving many machine learning problems, in which block-wise structure is commonly encountered. In this work, we propose cyclic block coordinate methods for nonconvex optimization problems with non-asymptotic gradient norm guarantees. Our convergence analysis is based on a gradient Lipschitz condition with respect to a Mahalanobis norm, inspired by a recent progress on cyclic block coordinate methods. In deterministic settings, our convergence guarantee matches the guarantee of (full-gradient) gradient descent, but with the gradient Lipschitz constant being defined w.r.t.~the Mahalanobis norm. In stochastic settings, we use recursive variance reduction to decrease the per-iteration cost and match the arithmetic operation complexity of current optimal stochastic full-gradient methods, with a unified analysis for both finite-sum and infinite-sum cases. We further prove the faster, linear convergence of our methods when a Polyak-{\L}ojasiewicz (P{\L}) condition holds for the objective function. To the best of our knowledge, our work is the first to provide variance-reduced convergence guarantees for a cyclic block coordinate method. Our experimental results demonstrate the efficacy of the proposed variance-reduced cyclic scheme in training deep neural nets.

Photo-identification (photo-id) is one of the main non-invasive capture-recapture methods utilised by marine researchers for monitoring cetacean (dolphin, whale, and porpoise) populations. This method has historically been performed manually resulting in high workload and cost due to the vast number of images collected. Recently automated aids have been developed to help speed-up photo-id, although they are often disjoint in their processing and do not utilise all available identifying information. Work presented in this paper aims to create a fully automatic photo-id aid capable of providing most likely matches based on all available information without the need for data pre-processing such as cropping. This is achieved through a pipeline of computer vision models and post-processing techniques aimed at detecting cetaceans in unedited field imagery before passing them downstream for individual level catalogue matching. The system is capable of handling previously uncatalogued individuals and flagging these for investigation thanks to catalogue similarity comparison. We evaluate the system against multiple real-life photo-id catalogues, achieving mAP@IOU[0.5] = 0.91, 0.96 for the task of dorsal fin detection on catalogues from Tanzania and the UK respectively and 83.1, 97.5% top-10 accuracy for the task of individual classification on catalogues from the UK and USA.

二重优化（BO）可用于解决各种重要的机器学习问题，包括但不限于超参数优化，元学习，持续学习和增强学习。常规的BO方法需要通过与隐式分化的低级优化过程进行区分，这需要与Hessian矩阵相关的昂贵计算。最近，人们一直在寻求BO的一阶方法，但是迄今为止提出的方法对于大规模的深度学习应用程序往往是复杂且不切实际的。在这项工作中，我们提出了一种简单的一阶BO算法，仅取决于一阶梯度信息，不需要隐含的区别，并且对于大规模的非凸函数而言是实用和有效的。我们为提出的方法提供了非注重方法分析非凸目标的固定点，并提出了表明其出色实践绩效的经验结果。

我们在大规模设置中研究一类广义的线性程序（GLP），包括可能简单的非光滑凸规律器和简单的凸集合约束。通过将GLP作为等效凸凹入最大问题的重新介绍，我们表明问题中的线性结构可用于设计高效，可扩展的一阶算法，我们给出了名称\ EMPH {坐标线性方差减少}（\ textsc {clvr};发音为``clever''）。 \ textsc {clvr}是一种增量坐标方法，具有隐式方差差异，输出双变量迭代的\ emph {仿射组合}。 \ textsc {clvr}产生改善的复杂性结果（glp），这取决于（glp）中的线性约束矩阵的最大行标准而不是光谱标准。当正常化术语和约束是可分离的，\ textsc {clvr}承认有效的延迟更新策略，使其复杂性界限与（glp）中的线性约束矩阵的非零元素的数量而不是矩阵尺寸。我们表明，通过引入稀疏连接的辅助变量，可以将基于$ F $ -divergence和Wassersein指标的歧义组的分布稳健优化（DRO）问题进行重新重整为（GLP）。我们补充了我们的理论保证，具有验证我们算法的实际效果的数值实验，无论是在壁钟时间和数据次数方面。

找到Reset中的参数的最佳配置是一个非凸显最小化问题，但一阶方法尽管如此，找到了过度分辨率制度的全局最优。通过将Reset的训练过程转化为梯度流部分微分方程（PDE）和检查该限制过程的收敛性能，我们研究了这种现象。假设激活函数为2美元 - 最佳或部分$ 1 $-homerence;正则Relu满足后一种条件。我们表明，如果Reset足够大，则深度和宽度根据代数上的准确性和置信水平，一阶优化方法可以找到适合培训数据的全局最小化器。

在一个拟合训练数据的深度神经网络（NN）中找到参数是一个非渗透优化问题，但基本的一阶优化方法（梯度下降）在许多实际情况下，具有完美拟合（零损失）的全局优化器。我们在限制性制度中检查残留神经网络（Reset）的剩余神经网络（Reset）的情况的这种现象，其中每个层（宽度）的层数（深度）和权重的数量均转到无穷大。首先，我们使用平均场限制参数来证明参数训练的梯度下降成为概率分布的梯度流，其特征在于大NN限制中的部分微分方程（PDE）。接下来，我们表明，在某些假设下，PDE的解决方案在训练时间内收敛到零损失解决方案。这些结果表明，如果Reset足够大，则reset的培训给出了近零损失。我们给出了减少给定阈值以下低于给定阈值的损失所需的深度和宽度的估计值。

我们研究了基于分布强大的机会约束的对抗性分类模型。我们表明，在Wasserstein模糊性下，该模型旨在最大限度地减少距离分类距离的条件值 - 风险，并且我们探讨了前面提出的对抗性分类模型和最大限度的分类机的链接。我们还提供了用于线性分类的分布鲁棒模型的重构，并且表明它相当于最小化正则化斜坡损失目标。数值实验表明，尽管这种配方的非凸起，但是标准的下降方法似乎会聚到全球最小值器。灵感来自这种观察，我们表明，对于某一类分布，正则化斜坡损失最小化问题的唯一静止点是全球最小化器。

Stochastic Gradient Descent (SGD) is a popular algorithm that can achieve state-of-the-art performance on a variety of machine learning tasks. Several researchers have recently proposed schemes to parallelize SGD, but all require performance-destroying memory locking and synchronization. This work aims to show using novel theoretical analysis, algorithms, and implementation that SGD can be implemented without any locking. We present an update scheme called Hogwild! which allows processors access to shared memory with the possibility of overwriting each other's work. We show that when the associated optimization problem is sparse, meaning most gradient updates only modify small parts of the decision variable, then Hogwild! achieves a nearly optimal rate of convergence. We demonstrate experimentally that Hogwild! outperforms alternative schemes that use locking by an order of magnitude.

With growing sophistication and volume of cyber attacks combined with complex network structures, it is becoming extremely difficult for security analysts to corroborate evidences to identify multistage campaigns on their network. This work develops HeAT (Heated Alert Triage): given a critical indicator of compromise (IoC), e.g., a severe IDS alert, HeAT produces a HeATed Attack Campaign (HAC) depicting the multistage activities that led up to the critical event. We define the concept of "Alert Episode Heat" to represent the analysts opinion of how much an event contributes to the attack campaign of the critical IoC given their knowledge of the network and security expertise. Leveraging a network-agnostic feature set, HeAT learns the essence of analyst's assessment of "HeAT" for a small set of IoC's, and applies the learned model to extract insightful attack campaigns for IoC's not seen before, even across networks by transferring what have been learned. We demonstrate the capabilities of HeAT with data collected in Collegiate Penetration Testing Competition (CPTC) and through collaboration with a real-world SOC. We developed HeAT-Gain metrics to demonstrate how analysts may assess and benefit from the extracted attack campaigns in comparison to common practices where IP addresses are used to corroborate evidences. Our results demonstrates the practical uses of HeAT by finding campaigns that span across diverse attack stages, remove a significant volume of irrelevant alerts, and achieve coherency to the analyst's original assessments.

Large language models (LLMs) have demonstrated impressive capabilities in natural language understanding and generation, but the quality bar for medical and clinical applications is high. Today, attempts to assess models' clinical knowledge typically rely on automated evaluations on limited benchmarks. There is no standard to evaluate model predictions and reasoning across a breadth of tasks. To address this, we present MultiMedQA, a benchmark combining six existing open question answering datasets spanning professional medical exams, research, and consumer queries; and HealthSearchQA, a new free-response dataset of medical questions searched online. We propose a framework for human evaluation of model answers along multiple axes including factuality, precision, possible harm, and bias. In addition, we evaluate PaLM (a 540-billion parameter LLM) and its instruction-tuned variant, Flan-PaLM, on MultiMedQA. Using a combination of prompting strategies, Flan-PaLM achieves state-of-the-art accuracy on every MultiMedQA multiple-choice dataset (MedQA, MedMCQA, PubMedQA, MMLU clinical topics), including 67.6% accuracy on MedQA (US Medical License Exam questions), surpassing prior state-of-the-art by over 17%. However, human evaluation reveals key gaps in Flan-PaLM responses. To resolve this we introduce instruction prompt tuning, a parameter-efficient approach for aligning LLMs to new domains using a few exemplars. The resulting model, Med-PaLM, performs encouragingly, but remains inferior to clinicians. We show that comprehension, recall of knowledge, and medical reasoning improve with model scale and instruction prompt tuning, suggesting the potential utility of LLMs in medicine. Our human evaluations reveal important limitations of today's models, reinforcing the importance of both evaluation frameworks and method development in creating safe, helpful LLM models for clinical applications.