Learning on Graphs (LoG) is widely used in multi-client systems when each client has insufficient local data, and multiple clients have to share their raw data to learn a model of good quality. One scenario is to recommend items to clients with limited historical data and sharing similar preferences with other clients in a social network. On the other hand, due to the increasing demands for the protection of clients' data privacy, Federated Learning (FL) has been widely adopted: FL requires models to be trained in a multi-client system and restricts sharing of raw data among clients. The underlying potential data-sharing conflict between LoG and FL is under-explored and how to benefit from both sides is a promising problem. In this work, we first formulate the Graph Federated Learning (GFL) problem that unifies LoG and FL in multi-client systems and then propose sharing hidden representation instead of the raw data of neighbors to protect data privacy as a solution. To overcome the biased gradient problem in GFL, we provide a gradient estimation method and its convergence analysis under the non-convex objective. In experiments, we evaluate our method in classification tasks on graphs. Our experiment shows a good match between our theory and the practice.

Pretrained large-scale vision-language models like CLIP have exhibited strong generalization over unseen tasks. Yet imperceptible adversarial perturbations can significantly reduce CLIP's performance on new tasks. In this work, we identify and explore the problem of \emph{adapting large-scale models for zero-shot adversarial robustness}. We first identify two key factors during model adaption -- training losses and adaptation methods -- that affect the model's zero-shot adversarial robustness. We then propose a text-guided contrastive adversarial training loss, which aligns the text embeddings and the adversarial visual features with contrastive learning on a small set of training data. We apply this training loss to two adaption methods, model finetuning and visual prompt tuning. We find that visual prompt tuning is more effective in the absence of texts, while finetuning wins in the existence of text guidance. Overall, our approach significantly improves the zero-shot adversarial robustness over CLIP, seeing an average improvement of over 31 points over ImageNet and 15 zero-shot datasets. We hope this work can shed light on understanding the zero-shot adversarial robustness of large-scale models.

Many visual recognition models are evaluated only on their classification accuracy, a metric for which they obtain strong performance. In this paper, we investigate whether computer vision models can also provide correct rationales for their predictions. We propose a ``doubly right'' object recognition benchmark, where the metric requires the model to simultaneously produce both the right labels as well as the right rationales. We find that state-of-the-art visual models, such as CLIP, often provide incorrect rationales for their categorical predictions. However, by transferring the rationales from language models into visual representations through a tailored dataset, we show that we can learn a ``why prompt,'' which adapts large visual representations to produce correct rationales. Visualizations and empirical experiments show that our prompts significantly improve performance on doubly right object recognition, in addition to zero-shot transfer to unseen tasks and datasets.

Deep networks for computer vision are not reliable when they encounter adversarial examples. In this paper, we introduce a framework that uses the dense intrinsic constraints in natural images to robustify inference. By introducing constraints at inference time, we can shift the burden of robustness from training to the inference algorithm, thereby allowing the model to adjust dynamically to each individual image's unique and potentially novel characteristics at inference time. Among different constraints, we find that equivariance-based constraints are most effective, because they allow dense constraints in the feature space without overly constraining the representation at a fine-grained level. Our theoretical results validate the importance of having such dense constraints at inference time. Our empirical experiments show that restoring feature equivariance at inference time defends against worst-case adversarial perturbations. The method obtains improved adversarial robustness on four datasets (ImageNet, Cityscapes, PASCAL VOC, and MS-COCO) on image recognition, semantic segmentation, and instance segmentation tasks. Project page is available at equi4robust.cs.columbia.edu.

Generic motion understanding from video involves not only tracking objects, but also perceiving how their surfaces deform and move. This information is useful to make inferences about 3D shape, physical properties and object interactions. While the problem of tracking arbitrary physical points on surfaces over longer video clips has received some attention, no dataset or benchmark for evaluation existed, until now. In this paper, we first formalize the problem, naming it tracking any point (TAP). We introduce a companion benchmark, TAP-Vid, which is composed of both real-world videos with accurate human annotations of point tracks, and synthetic videos with perfect ground-truth point tracks. Central to the construction of our benchmark is a novel semi-automatic crowdsourced pipeline which uses optical flow estimates to compensate for easier, short-term motion like camera shake, allowing annotators to focus on harder sections of video. We validate our pipeline on synthetic data and propose a simple end-to-end point tracking model TAP-Net, showing that it outperforms all prior methods on our benchmark when trained on synthetic data.

人的大脑位于复杂的神经生物学系统的核心，神经元，电路和子系统以神秘的方式相互作用。长期以来，了解大脑的结构和功能机制一直是神经科学研究和临床障碍疗法的引人入胜的追求。将人脑作为网络的连接映射是神经科学中最普遍的范例之一。图神经网络（GNN）最近已成为建模复杂网络数据的潜在方法。另一方面，深层模型的可解释性低，从而阻止了他们在医疗保健等决策环境中的使用。为了弥合这一差距，我们提出了一个可解释的框架，以分析特定的利益区域（ROI）和突出的联系。提出的框架由两个模块组成：疾病预测的面向脑网络的主链模型和全球共享的解释发生器，该模型突出了包括疾病特异性的生物标志物，包括显着的ROI和重要连接。我们在三个现实世界中的脑疾病数据集上进行实验。结果证明了我们的框架可以获得出色的性能并确定有意义的生物标志物。这项工作的所有代码均可在https://github.com/hennyjie/ibgnn.git上获得。

大脑网络将大脑区域之间的复杂连接性描述为图形结构，这为研究脑连接素提供了强大的手段。近年来，图形神经网络已成为使用结构化数据的普遍学习范式。但是，由于数据获取的成本相对较高，大多数大脑网络数据集的样本量受到限制，这阻碍了足够的培训中的深度学习模型。受元学习的启发，该论文以有限的培训示例快速学习新概念，研究了在跨数据库中分析脑连接组的数据有效培训策略。具体而言，我们建议在大型样本大小的数据集上进行元训练模型，并将知识转移到小数据集中。此外，我们还探索了两种面向脑网络的设计，包括Atlas转换和自适应任务重新启动。与其他训练前策略相比，我们的基于元学习的方法实现了更高和稳定的性能，这证明了我们提出的解决方案的有效性。该框架还能够以数据驱动的方式获得有关数据集和疾病之间相似之处的新见解。

图形神经网络（GNN）已被广泛用于建模图形结构化数据，这是由于其在广泛的实用应用中令人印象深刻的性能。最近，GNNS的知识蒸馏（KD）在图形模型压缩和知识转移方面取得了显着进步。但是，大多数现有的KD方法都需要大量的真实数据，这些数据在实践中不容易获得，并且可能排除其在教师模型对稀有或难以获取数据集培训的情况下的适用性。为了解决这个问题，我们提出了第一个用于图形结构化数据（DFAD-GNN）的无数据对抗知识蒸馏的端到端框架。具体而言，我们的DFAD-GNN采用生成性对抗网络，主要由三个组成部分组成：预先训练的教师模型和学生模型被视为两个歧视者，并利用生成器来衍生训练图来从教师模型进入学生模型。在各种基准模型和六个代表性数据集上进行的广泛实验表明，我们的DFAD-GNN在图形分类任务中显着超过了最新的无数据基线。

Mapping the connectome of the human brain using structural or functional connectivity has become one of the most pervasive paradigms for neuroimaging analysis. Recently, Graph Neural Networks (GNNs) motivated from geometric deep learning have attracted broad interest due to their established power for modeling complex networked data. Despite their superior performance in many fields, there has not yet been a systematic study of how to design effective GNNs for brain network analysis. To bridge this gap, we present BrainGB, a benchmark for brain network analysis with GNNs. BrainGB standardizes the process by (1) summarizing brain network construction pipelines for both functional and structural neuroimaging modalities and (2) modularizing the implementation of GNN designs. We conduct extensive experiments on datasets across cohorts and modalities and recommend a set of general recipes for effective GNN designs on brain networks. To support open and reproducible research on GNN-based brain network analysis, we host the BrainGB website at https://braingb.us with models, tutorials, examples, as well as an out-of-box Python package. We hope that this work will provide useful empirical evidence and offer insights for future research in this novel and promising direction.

图形神经网络（GNNS），作为一组强大的表示对不规则数据学习的强大工具，在各种下游任务中表现出优越性。具有表示为概念映射的非结构化文本，可以针对文档检索等任务来利用GNN。呼吸GNNS如何帮助文档检索，我们对大型多学科数据集电源线19进行实证研究。结果表明，我们提出的语义导向图函数的基于BM25检索的候选人，而不是杜松子酒和GAT等复杂的结构导向GNN，而不是杜松子酒和GATS，而不是基于BM25检索到的候选者实现更好且更稳定的性能。我们在本案例研究中的见解可以作为未来工作的指导准则，以便为文档检索和分类等文本推理任务提供适当的语义导向的归纳偏差。此案例研究的所有代码都可以在https://github.com/hennyjie/gnn-docrocrocal中获得。