Some recent works observed the instability of post-hoc explanations when input side perturbations are applied to the model. This raises the interest and concern in the stability of post-hoc explanations. However, the remaining question is: is the instability caused by the neural network model or the post-hoc explanation method? This work explores the potential source that leads to unstable post-hoc explanations. To separate the influence from the model, we propose a simple output probability perturbation method. Compared to prior input side perturbation methods, the output probability perturbation method can circumvent the neural model's potential effect on the explanations and allow the analysis on the explanation method. We evaluate the proposed method with three widely-used post-hoc explanation methods (LIME (Ribeiro et al., 2016), Kernel Shapley (Lundberg and Lee, 2017a), and Sample Shapley (Strumbelj and Kononenko, 2010)). The results demonstrate that the post-hoc methods are stable, barely producing discrepant explanations under output probability perturbations. The observation suggests that neural network models may be the primary source of fragile explanations.

Understanding why a model makes certain predictions is crucial when adapting it for real world decision making. LIME is a popular model-agnostic feature attribution method for the tasks of classification and regression. However, the task of learning to rank in information retrieval is more complex in comparison with either classification or regression. In this work, we extend LIME to propose Rank-LIME, a model-agnostic, local, post-hoc linear feature attribution method for the task of learning to rank that generates explanations for ranked lists. We employ novel correlation-based perturbations, differentiable ranking loss functions and introduce new metrics to evaluate ranking based additive feature attribution models. We compare Rank-LIME with a variety of competing systems, with models trained on the MS MARCO datasets and observe that Rank-LIME outperforms existing explanation algorithms in terms of Model Fidelity and Explain-NDCG. With this we propose one of the first algorithms to generate additive feature attributions for explaining ranked lists.

越来越多的电子健康记录（EHR）数据和深度学习技术进步的越来越多的可用性（DL）已经引发了在开发基于DL的诊断，预后和治疗的DL临床决策支持系统中的研究兴趣激增。尽管承认医疗保健的深度学习的价值，但由于DL的黑匣子性质，实际医疗环境中进一步采用的障碍障碍仍然存在。因此，有一个可解释的DL的新兴需求，它允许最终用户评估模型决策，以便在采用行动之前知道是否接受或拒绝预测和建议。在这篇综述中，我们专注于DL模型在医疗保健中的可解释性。我们首先引入深入解释性的方法，并作为该领域的未来研究人员或临床从业者的方法参考。除了这些方法的细节之外，我们还包括对这些方法的优缺点以及它们中的每个场景都适合的讨论，因此感兴趣的读者可以知道如何比较和选择它们供使用。此外，我们讨论了这些方法，最初用于解决一般域问题，已经适应并应用于医疗保健问题以及如何帮助医生更好地理解这些数据驱动技术。总的来说，我们希望这项调查可以帮助研究人员和从业者在人工智能（AI）和临床领域了解我们为提高其DL模型的可解释性并相应地选择最佳方法。

众所周知，端到端的神经NLP体系结构很难理解，这引起了近年来为解释性建模的许多努力。模型解释的基本原则是忠诚，即，解释应准确地代表模型预测背后的推理过程。这项调查首先讨论了忠诚的定义和评估及其对解释性的意义。然后，我们通过将方法分为五类来介绍忠实解释的最新进展：相似性方法，模型内部结构的分析，基于反向传播的方法，反事实干预和自我解释模型。每个类别将通过其代表性研究，优势和缺点来说明。最后，我们从它们的共同美德和局限性方面讨论了上述所有方法，并反思未来的工作方向忠实的解释性。对于有兴趣研究可解释性的研究人员，这项调查将为该领域提供可访问且全面的概述，为进一步探索提供基础。对于希望更好地了解自己的模型的用户，该调查将是一项介绍性手册，帮助选择最合适的解释方法。

估算预测预测的语言模型的不确定性对于提高NLP的可靠性是重要的。虽然许多以前的作品侧重于量化预测不确定性，但在解释不确定性时几乎没有工作。本文进一步推动了一个关于解释后校准的预训练的语言模型的不确定预测。我们适应了两种基于扰动的后宫释放方法，留出次出来和采样福利，以识别引起预测中不确定性的输入中的单词。我们以三项任务测试BERT和Roberta上提出的方法：情绪分类，自然语言推断和解释域，在域内和域外设置。实验表明，两种方法都始终捕获引起预测不确定性的输入中的单词。

Deep Learning and Machine Learning based models have become extremely popular in text processing and information retrieval. However, the non-linear structures present inside the networks make these models largely inscrutable. A significant body of research has focused on increasing the transparency of these models. This article provides a broad overview of research on the explainability and interpretability of natural language processing and information retrieval methods. More specifically, we survey approaches that have been applied to explain word embeddings, sequence modeling, attention modules, transformers, BERT, and document ranking. The concluding section suggests some possible directions for future research on this topic.

可解释的机器学习提供了有关哪些因素推动了黑盒系统的一定预测以及是否信任高风险决策或大规模部署的洞察力。现有方法主要集中于选择解释性输入功能，这些功能遵循本地添加剂或实例方法。加性模型使用启发式采样扰动来依次学习实例特定解释器。因此，该过程效率低下，并且容易受到条件较差的样品的影响。同时，实例技术直接学习本地采样分布，并可以从其他输入中利用全球信息。但是，由于严格依赖预定义的功能，他们只能解释单一级预测并在不同设置上遇到不一致的情况。这项工作利用了这两种方法的优势，并提出了一个全球框架，用于同时学习多个目标类别的本地解释。我们还提出了一种自适应推理策略，以确定特定实例的最佳功能数量。我们的模型解释器极大地超过了忠诚的添加和实例的对应物，而在各种数据集和Black-box模型体系结构上获得了高水平的简洁性。

Explainability has been widely stated as a cornerstone of the responsible and trustworthy use of machine learning models. With the ubiquitous use of Deep Neural Network (DNN) models expanding to risk-sensitive and safety-critical domains, many methods have been proposed to explain the decisions of these models. Recent years have also seen concerted efforts that have shown how such explanations can be distorted (attacked) by minor input perturbations. While there have been many surveys that review explainability methods themselves, there has been no effort hitherto to assimilate the different methods and metrics proposed to study the robustness of explanations of DNN models. In this work, we present a comprehensive survey of methods that study, understand, attack, and defend explanations of DNN models. We also present a detailed review of different metrics used to evaluate explanation methods, as well as describe attributional attack and defense methods. We conclude with lessons and take-aways for the community towards ensuring robust explanations of DNN model predictions.

Interpretability provides a means for humans to verify aspects of machine learning (ML) models and empower human+ML teaming in situations where the task cannot be fully automated. Different contexts require explanations with different properties. For example, the kind of explanation required to determine if an early cardiac arrest warning system is ready to be integrated into a care setting is very different from the type of explanation required for a loan applicant to help determine the actions they might need to take to make their application successful. Unfortunately, there is a lack of standardization when it comes to properties of explanations: different papers may use the same term to mean different quantities, and different terms to mean the same quantity. This lack of a standardized terminology and categorization of the properties of ML explanations prevents us from both rigorously comparing interpretable machine learning methods and identifying what properties are needed in what contexts. In this work, we survey properties defined in interpretable machine learning papers, synthesize them based on what they actually measure, and describe the trade-offs between different formulations of these properties. In doing so, we enable more informed selection of task-appropriate formulations of explanation properties as well as standardization for future work in interpretable machine learning.

A critical problem in post hoc explainability is the lack of a common foundational goal among methods. For example, some methods are motivated by function approximation, some by game theoretic notions, and some by obtaining clean visualizations. This fragmentation of goals causes not only an inconsistent conceptual understanding of explanations but also the practical challenge of not knowing which method to use when. In this work, we begin to address these challenges by unifying eight popular post hoc explanation methods (LIME, C-LIME, SHAP, Occlusion, Vanilla Gradients, Gradients x Input, SmoothGrad, and Integrated Gradients). We show that these methods all perform local function approximation of the black-box model, differing only in the neighbourhood and loss function used to perform the approximation. This unification enables us to (1) state a no free lunch theorem for explanation methods which demonstrates that no single method can perform optimally across all neighbourhoods, and (2) provide a guiding principle to choose among methods based on faithfulness to the black-box model. We empirically validate these theoretical results using various real-world datasets, model classes, and prediction tasks. By bringing diverse explanation methods into a common framework, this work (1) advances the conceptual understanding of these methods, revealing their shared local function approximation objective, properties, and relation to one another, and (2) guides the use of these methods in practice, providing a principled approach to choose among methods and paving the way for the creation of new ones.

注意机制主导着深层模型的解释性。它们在输入上产生概率分布，该输入被广泛认为是特征对重要指标。但是，在本文中，我们发现注意力解释中的一个关键局限性：识别特征影响的极性的弱点。这将是一种误导性 - 注意力较高的特征可能不会忠实地促进模型预测；相反，它们可以施加抑制作用。有了这一发现，我们反思了当前基于注意力的技术的解释性，例如Attentio $ \ odot $梯度和基于LRP的注意解释。我们首先提出了一种可操作的诊断方法（此后忠实违规测试），以衡量解释权重与影响极性之间的一致性。通过广泛的实验，我们表明大多数经过测试的解释方法出乎意料地受到违反忠诚问题的阻碍，尤其是原始关注。对影响违规问题的因素的经验分析进一步为采用注意模型中采用解释方法提供了有用的观察。

As machine learning black boxes are increasingly being deployed in domains such as healthcare and criminal justice, there is growing emphasis on building tools and techniques for explaining these black boxes in an interpretable manner. Such explanations are being leveraged by domain experts to diagnose systematic errors and underlying biases of black boxes. In this paper, we demonstrate that post hoc explanations techniques that rely on input perturbations, such as LIME and SHAP, are not reliable. Specifically, we propose a novel scaffolding technique that effectively hides the biases of any given classifier by allowing an adversarial entity to craft an arbitrary desired explanation. Our approach can be used to scaffold any biased classifier in such a way that its predictions on the input data distribution still remain biased, but the post hoc explanations of the scaffolded classifier look innocuous. Using extensive evaluation with multiple real world datasets (including COMPAS), we demonstrate how extremely biased (racist) classifiers crafted by our framework can easily fool popular explanation techniques such as LIME and SHAP into generating innocuous explanations which do not reflect the underlying biases. CCS CONCEPTS• Computing methodologies → Machine learning; Supervised learning by classification; • Human-centered computing → Interactive systems and tools.

对AI模型用于支持网络安全决策的预测的解释至关重要。尤其如此，当模型的错误预测可能导致严重损失甚至对生命和关键资产造成损失时。但是，尽管在大多数情况下表现出色，但大多数现有的AI模型都无法对其预测结果提供解释。在这项工作中，我们提出了一种新颖的可解释的AI方法，称为Philaex，该方法提供了启发式手段，以识别优化的特征子集，以形成对AI模型预测的完整解释。它标识了导致模型的边界预测的功能，并提取了具有积极贡献的人。然后，通过优化脊回归模型来量化特征归因。我们通过两个实验来验证解释保真度。首先，我们通过Philaex的特征归因值评估了我们的方法在正确识别Android Malwares对抗样本中激活特征的能力。其次，扣除和增强测试用于评估解释的保真度。结果表明，与最先进的方法（例如石灰和摇动​​）相比，Philaex能够正确解释不同类型的分类器，并具有更高的保真度解释。

Explainable artificial intelligence is proposed to provide explanations for reasoning performed by an Artificial Intelligence. There is no consensus on how to evaluate the quality of these explanations, since even the definition of explanation itself is not clear in the literature. In particular, for the widely known Local Linear Explanations, there are qualitative proposals for the evaluation of explanations, although they suffer from theoretical inconsistencies. The case of image is even more problematic, where a visual explanation seems to explain a decision while detecting edges is what it really does. There are a large number of metrics in the literature specialized in quantitatively measuring different qualitative aspects so we should be able to develop metrics capable of measuring in a robust and correct way the desirable aspects of the explanations. In this paper, we propose a procedure called REVEL to evaluate different aspects concerning the quality of explanations with a theoretically coherent development. This procedure has several advances in the state of the art: it standardizes the concepts of explanation and develops a series of metrics not only to be able to compare between them but also to obtain absolute information regarding the explanation itself. The experiments have been carried out on image four datasets as benchmark where we show REVEL's descriptive and analytical power.

State-of-the-art machine translation evaluation metrics are based on black-box language models. Hence, recent works consider their explainability with the goals of better understandability for humans and better metric analysis, including failure cases. In contrast, we explicitly leverage explanations to boost the metrics' performance. In particular, we perceive explanations as word-level scores, which we convert, via power means, into sentence-level scores. We combine this sentence-level score with the original metric to obtain a better metric. Our extensive evaluation and analysis across 5 datasets, 5 metrics and 4 explainability techniques shows that some configurations reliably improve the original metrics' correlation with human judgment. On two held datasets for testing, we obtain improvements in 15/18 resp. 4/4 cases. The gains in Pearson correlation are up to 0.032 resp. 0.055. We make our code available.

本文解决了解释黑框回归模型异常预测的任务。当使用黑框模型（例如从许多传感器测量值中预测能源消耗的一个模型）时，我们通常会有某些观察到的样品可能会显着偏离其预测的情况。这可能是由于亚最佳黑盒模型，或仅仅​​是因为这些样品是异常值。无论哪种情况，理想情况下都希望计算``责任分数''，以指示输入变量负责异常输出的程度。在这项工作中，我们将此任务形式化为一个统计逆问题：给定模型偏离预期值，推断每个输入变量的责任分数。我们提出了一种称为似然补偿（LC）的新方法，该方法基于可能性原理，并计算对每个输入变量的校正。据我们所知，这是第一个计算实际有价值异常模型偏差的责任分数的原则性框架。我们将方法应用于现实世界中的建筑能源预测任务，并根据专家反馈确认其实用性。

在人类循环机器学习应用程序的背景下，如决策支持系统，可解释性方法应在不使用户等待的情况下提供可操作的见解。在本文中，我们提出了加速的模型 - 不可知论解释（ACME），一种可解释的方法，即在全球和本地层面迅速提供特征重要性分数。可以将acme应用于每个回归或分类模型的后验。 ACME计算功能排名不仅提供了一个什么，但它还提供了一个用于评估功能值的变化如何影响模型预测的原因 - 如果分析工具。我们评估了综合性和现实世界数据集的建议方法，同时也与福芙添加剂解释（Shap）相比，我们制作了灵感的方法，目前是最先进的模型无关的解释性方法。我们在生产解释的质量方面取得了可比的结果，同时急剧减少计算时间并为全局和局部解释提供一致的可视化。为了促进该领域的研究，为重复性，我们还提供了一种存储库，其中代码用于实验。

了解神经网络的决策过程很难。解释的一种重要方法是将其决定归因于关键特征。尽管提出了许多算法，但其中大多数仅改善了模型的忠诚。但是，真实的环境包含许多随机噪声，这可能会导致解释中的波动。更严重的是，最近的作品表明，解释算法容易受到对抗性攻击的影响。所有这些使解释很难在实际情况下信任。为了弥合这一差距，我们提出了一种模型 - 不稳定方法\ emph {特征归因}（METFA）的中位数测试，以量化不确定性并提高使用理论保证的解释算法的稳定性。 METFA具有以下两个函数：（1）检查一个特征是显着重要还是不重要，并生成METFA相关的映射以可视化结果； （2）计算特征归因评分的置信区间，并生成一个平滑的图表以提高解释的稳定性。实验表明，METFA提高了解释的视觉质量，并在保持忠诚的同时大大减少了不稳定。为了定量评估不同噪音设置下解释的忠诚，我们进一步提出了几个强大的忠诚指标。实验结果表明，METFA平滑的解释可以显着提高稳健的忠诚。此外，我们使用两种方案来显示METFA在应用程序中的潜力。首先，当应用于SOTA解释方法来定位语义分割模型的上下文偏见时，METFA很重要的解释使用较小的区域来维持99 \％+忠实。其次，当通过不同的以解释为导向的攻击进行测试时，METFA可以帮助捍卫香草，以及自适应的对抗性攻击，以防止解释。

最先进的实体匹配（EM）方法很难解释，并且为EM带来可解释的AI具有重要的价值。不幸的是，大多数流行的解释性方法无法开箱即用，需要适应。在本文中，我们确定了将本地事后特征归因方法应用于实体匹配的三个挑战：跨记录的交互作用，不匹配的解释和灵敏度变化。我们提出了新颖的模型 - 静态和模式 - 富含模型的方法柠檬柠檬，该方法通过（i）产生双重解释来避免交叉记录的互动效果来应对所有三个挑战，（ii）介绍了归因潜力的新颖概念，以解释两个记录如何能够拥有如何具有匹配，（iii）自动选择解释粒度以匹配匹配器和记录对的灵敏度。公共数据集上的实验表明，所提出的方法更忠实于匹配器，并且在帮助用户了解匹配器的决策边界的工作中比以前的工作更具忠诚度。此外，用户研究表明，与标准的解释相比石灰的适应。

深层神经网络以其对各种机器学习和人工智能任务的精湛处理而闻名。但是，由于其过度参数化的黑盒性质，通常很难理解深层模型的预测结果。近年来，已经提出了许多解释工具来解释或揭示模型如何做出决策。在本文中，我们回顾了这一研究，并尝试进行全面的调查。具体来说，我们首先介绍并阐明了人们通常会感到困惑的两个基本概念 - 解释和解释性。为了解决解释中的研究工作，我们通过提出新的分类法来阐述许多解释算法的设计。然后，为了了解解释结果，我们还调查了评估解释算法的性能指标。此外，我们总结了使用“可信赖”解释算法评估模型的解释性的当前工作。最后，我们审查并讨论了深层模型的解释与其他因素之间的联系，例如对抗性鲁棒性和从解释中学习，并介绍了一些开源库，以解释算法和评估方法。