Recent trends in language modeling have focused on increasing performance through scaling, and have resulted in an environment where training language models is out of reach for most researchers and practitioners. While most in the community are asking how to push the limits of extreme computation, we ask the opposite question: How far can we get with a single GPU in just one day? We investigate the downstream performance achievable with a transformer-based language model trained completely from scratch with masked language modeling for a single day on a single consumer GPU. Aside from re-analyzing nearly all components of the pretraining pipeline for this scenario and providing a modified pipeline with performance close to BERT, we investigate why scaling down is hard, and which modifications actually improve performance in this scenario. We provide evidence that even in this constrained setting, performance closely follows scaling laws observed in large-compute settings. Through the lens of scaling laws, we categorize a range of recent improvements to training and architecture and discuss their merit and practical applicability (or lack thereof) for the limited compute setting.

Vision transformers (ViTs) are quickly becoming the de-facto architecture for computer vision, yet we understand very little about why they work and what they learn. While existing studies visually analyze the mechanisms of convolutional neural networks, an analogous exploration of ViTs remains challenging. In this paper, we first address the obstacles to performing visualizations on ViTs. Assisted by these solutions, we observe that neurons in ViTs trained with language model supervision (e.g., CLIP) are activated by semantic concepts rather than visual features. We also explore the underlying differences between ViTs and CNNs, and we find that transformers detect image background features, just like their convolutional counterparts, but their predictions depend far less on high-frequency information. On the other hand, both architecture types behave similarly in the way features progress from abstract patterns in early layers to concrete objects in late layers. In addition, we show that ViTs maintain spatial information in all layers except the final layer. In contrast to previous works, we show that the last layer most likely discards the spatial information and behaves as a learned global pooling operation. Finally, we conduct large-scale visualizations on a wide range of ViT variants, including DeiT, CoaT, ConViT, PiT, Swin, and Twin, to validate the effectiveness of our method.

Cutting-edge diffusion models produce images with high quality and customizability, enabling them to be used for commercial art and graphic design purposes. But do diffusion models create unique works of art, or are they stealing content directly from their training sets? In this work, we study image retrieval frameworks that enable us to compare generated images with training samples and detect when content has been replicated. Applying our frameworks to diffusion models trained on multiple datasets including Oxford flowers, Celeb-A, ImageNet, and LAION, we discuss how factors such as training set size impact rates of content replication. We also identify cases where diffusion models, including the popular Stable Diffusion model, blatantly copy from their training data.

Facial analysis systems have been deployed by large companies and critiqued by scholars and activists for the past decade. Many existing algorithmic audits examine the performance of these systems on later stage elements of facial analysis systems like facial recognition and age, emotion, or perceived gender prediction; however, a core component to these systems has been vastly understudied from a fairness perspective: face detection, sometimes called face localization. Since face detection is a pre-requisite step in facial analysis systems, the bias we observe in face detection will flow downstream to the other components like facial recognition and emotion prediction. Additionally, no prior work has focused on the robustness of these systems under various perturbations and corruptions, which leaves open the question of how various people are impacted by these phenomena. We present the first of its kind detailed benchmark of face detection systems, specifically examining the robustness to noise of commercial and academic models. We use both standard and recently released academic facial datasets to quantitatively analyze trends in face detection robustness. Across all the datasets and systems, we generally find that photos of individuals who are $\textit{masculine presenting}$, $\textit{older}$, of $\textit{darker skin type}$, or have $\textit{dim lighting}$ are more susceptible to errors than their counterparts in other identities.

预训练的视觉模型（例如，剪辑）在许多下游任务中显示出有希望的零弹性概括，并具有正确设计的文本提示。最近的作品不依赖手工设计的提示，而是使用下游任务的培训数据来学习提示。虽然有效，但针对领域数据的培训却降低了模型的概括能力，使其无法看到新领域。在这项工作中，我们提出了测试时间提示调整（TPT），该方法可以通过单个测试样本即时学习自适应提示。对于图像分类，TPT通过使用置信度选择最小化熵来优化提示，以便模型在每个测试样本的不同增强视图上都具有一致的预测。在评估对自然分布变化的概括时，TPT平均将零击的TOP-1精度提高了3.6％，超过了先前需要其他特定于任务的训练数据的迅速调整方法。在评估看不见类别的跨数据集泛化时，TPT与使用其他培训数据的最先进方法相当。项目页面：https：//azshue.github.io/tpt。

标准扩散模型涉及图像变换 - 添加高斯噪声 - 以及逆转此降解的图像恢复操作员。我们观察到，扩散模型的生成行为并不是很大程度上取决于图像降解的选择，实际上，可以通过改变这种选择来构建整个生成模型家族。即使使用完全确定性的降解（例如，模糊，掩蔽等），培训和测试时间更新规则是基于扩散模型的培训和测试时间更新规则，可以轻松地概括为创建生成模型。这些完全确定的模型的成功使社区对扩散模型的理解质疑，这依赖于梯度Langevin动力学或变异推理中的噪声，并为反转任意过程的广义扩散模型铺平了道路。我们的代码可从https://github.com/arpitbansal297/cold-diffusion-models获得

水印是保护创作者对数字图像，视频和音频的权利的常用策略。最近，水印方法已扩展到深度学习模型 - 原则上，当对手试图复制该模型时，应保留水印。但是，实际上，智能对手通常可以去除水印。几篇论文提出了水印方法，这些方法声称对不同类型的拆除攻击具有耐药性，但是在面对新的或更好的对手时，这些新技术通常会失败。在本文中，我们提出了一种可认证的水印方法。使用Chiang等人提出的随机平滑技术，我们表明我们的水印是不明显的，除非模型参数的更改超过一定的L2阈值。除了获得认证外，与以前的水印方法相比，我们的水印在经验上也更强。我们的实验可以在https://github.com/arpitbansal297/certified_watermarks上复制。

对表格数据的深度学习的最新工作表明了深层表格模型的强劲表现，通常会弥合梯度增强的决策树和神经网络之间的差距。除了准确性之外，神经模型的主要优点是它们学习可重复使用的功能，并且在新域中很容易进行微调。该属性通常在计算机视觉和自然语言应用中被利用，在特定于任务的培训数据稀缺时，转移学习是必不可少的。在这项工作中，我们证明上游数据使表格神经网络比广泛使用的GBDT模型具有决定性的优势。我们为表格转移学习提出了一个现实的医学诊断基准，并提出了使用上游数据来通过各种表格神经网络体系结构来提高性能的方法指南。最后，我们为上游和下游特征集不同的情况提出了一种伪特征方法，在现实世界中，特定于表格的问题广泛。我们的代码可在https://github.com/levinroman/tabular-transfer-learning上找到。

具有数值节点特征和图形结构的图形神经网络（GNNS）作为输入显示出具有图形数据的各种监督学习任务的卓越性能。但是，GNN使用的数值节点特征通常是从大多数真实世界应用中的文本或表格（数字/分类）类型的原始数据中提取的。在大多数标准监督的学习设置中，使用IID（NON-GRAPH）数据的最佳模型不是简单的神经网络层，因此不容易被纳入GNN。在这里，我们提出了一个强大的堆叠框架，该框架将图形感知的传播与用于IID数据的任意模型融合在一起，这些模型是在多层中结合并堆叠的。我们的层面框架利用行李和堆叠策略来享受强有力的概括，从而有效地减轻了标签泄漏和过度拟合的方式。在各种具有表格/文本节点特征的图形数据集中，我们的方法相对于表格/文本和图形神经网络模型以及将两者结合的现有最新混合策略而获得了可比性或卓越的性能。

从社交媒体中刮擦的数据的流行率是获取数据集的一种手段，这导致人们对未经授权使用数据的关注日益严重。已经提出了数据中毒攻击是一种反对刮擦的堡垒，因为它们通过添加微小的，不可察觉的扰动来使数据“无法透视”。不幸的是，现有方法需要了解目标体系结构和完整的数据集，以便可以训练替代网络，其参数用于生成攻击。在这项工作中，我们引入了自回旋（AR）中毒，这种方法可以生成中毒的数据而无需访问更广泛的数据集。提出的AR扰动是通用的，可以在不同的数据集上应用，并且可以毒化不同的体系结构。与现有的未透视方法相比，我们的AR毒物更具抵抗力的防御能力，例如对抗性训练和强大的数据增强。我们的分析进一步洞悉了有效的数据毒物。