The dual-encoder has become the de facto architecture for dense retrieval. Typically, it computes the latent representations of the query and document independently, thus failing to fully capture the interactions between the query and document. To alleviate this, recent work expects to get query-informed representations of documents. During training, it expands the document with a real query, while replacing the real query with a generated pseudo query at inference. This discrepancy between training and inference makes the dense retrieval model pay more attention to the query information but ignore the document when computing the document representation. As a result, it even performs worse than the vanilla dense retrieval model, since its performance depends heavily on the relevance between the generated queries and the real query. In this paper, we propose a curriculum sampling strategy, which also resorts to the pseudo query at training and gradually increases the relevance of the generated query to the real query. In this way, the retrieval model can learn to extend its attention from the document only to both the document and query, hence getting high-quality query-informed document representations. Experimental results on several passage retrieval datasets show that our approach outperforms the previous dense retrieval methods1.

在本文中，我们提出了一个新的密集检索模型，该模型通过深度查询相互作用学习了各种文档表示。我们的模型使用一组生成的伪Queries编码每个文档，以获取查询信息的多视文档表示。它不仅具有较高的推理效率，例如《香草双编码模型》，而且还可以在文档编码中启用深度查询文档的交互，并提供多方面的表示形式，以更好地匹配不同的查询。几个基准的实验证明了所提出的方法的有效性，表现出色的双重编码基准。

This paper presents a pre-training technique called query-as-context that uses query prediction to improve dense retrieval. Previous research has applied query prediction to document expansion in order to alleviate the problem of lexical mismatch in sparse retrieval. However, query prediction has not yet been studied in the context of dense retrieval. Query-as-context pre-training assumes that the predicted query is a special context for the document and uses contrastive learning or contextual masked auto-encoding learning to compress the document and query into dense vectors. The technique is evaluated on large-scale passage retrieval benchmarks and shows considerable improvements compared to existing strong baselines such as coCondenser and CoT-MAE, demonstrating its effectiveness. Our code will be available at https://github.com/caskcsg/ir/tree/main/cotmae-qc .

当前的密集文本检索模型面临两个典型的挑战。首先，他们采用暹罗双重编码架构来独立编码查询和文档，以快速索引和搜索，同时忽略了较细粒度的术语互动。这导致了次优的召回表现。其次，他们的模型培训高度依赖于负面抽样技术，以在其对比损失中构建负面文档。为了应对这些挑战，我们提出了对抗猎犬速率（AR2），它由双重编码猎犬加上跨编码器等级组成。这两种模型是根据最小群体对手的共同优化的：检索员学会了检索负面文件以欺骗排名者，而排名者学会了对包括地面和检索的候选人进行排名，并提供渐进的直接反馈对双编码器检索器。通过这款对抗性游戏，猎犬逐渐生产出更难的负面文件来训练更好的排名者，而跨编码器排名者提供了渐进式反馈以改善检索器。我们在三个基准测试基准上评估AR2。实验结果表明，AR2始终如一地胜过现有的致密回收者方法，并在所有这些方法上实现了新的最新结果。这包括对自然问题的改进R@5％至77.9％（+2.1％），Triviaqa R@5％至78.2％（+1.4）和MS-Marco MRR@10％至39.5％（+1.3％）。代码和型号可在https://github.com/microsoft/ar2上找到。

我们提出了一种以最小计算成本提高广泛检索模型的性能的框架。它利用由基本密度检索方法提取的预先提取的文档表示，并且涉及训练模型以共同评分每个查询的一组检索到的候选文档，同时在其他候选的上下文中暂时转换每个文档的表示。以及查询本身。当基于其与查询的相似性进行评分文档表示时，该模型因此意识到其“对等”文档的表示。我们表明，我们的方法导致基本方法的检索性能以及彼此隔离的评分候选文档进行了大量改善，如在一对培训环境中。至关重要的是，与基于伯特式编码器的术语交互重型器不同，它在运行时在任何第一阶段方法的顶部引发可忽略不计的计算开销，允许它与任何最先进的密集检索方法容易地结合。最后，同时考虑给定查询的一组候选文档，可以在检索中进行额外的有价值的功能，例如评分校准和减轻排名中的社会偏差。

已经表明，在一个域上训练的双编码器经常概括到其他域以获取检索任务。一种广泛的信念是，一个双编码器的瓶颈层，其中最终得分仅仅是查询向量和通道向量之间的点产品，它过于局限，使得双编码器是用于域外概括的有效检索模型。在本文中，我们通过缩放双编码器模型的大小{\ em同时保持固定的瓶颈嵌入尺寸固定的瓶颈的大小来挑战这一信念。令人惊讶的是，令人惊讶的是，缩放模型尺寸会对各种缩放提高检索任务，特别是对于域外泛化。实验结果表明，我们的双编码器，\ textbf {g} enovalizable \ textbf {t} eTrievers（gtr），优先级％colbert〜\ cite {khattab2020colbertt}和现有的稀疏和密集的索取Beir DataSet〜\ Cite {Thakur2021Beir}显着显着。最令人惊讶的是，我们的消融研究发现，GTR是非常数据的高效，因为它只需要10 \％MARCO监督数据，以实现最佳域的性能。所有GTR模型都在https://tfhub.dev/google/collections/gtr/1发布。

To improve the performance of the dual-encoder retriever, one effective approach is knowledge distillation from the cross-encoder ranker. Existing works construct the candidate passages following the supervised learning setting where a query is paired with a positive passage and a batch of negatives. However, through empirical observation, we find that even the hard negatives from advanced methods are still too trivial for the teacher to distinguish, preventing the teacher from transferring abundant dark knowledge to the student through its soft label. To alleviate this issue, we propose ADAM, a knowledge distillation framework that can better transfer the dark knowledge held in the teacher with Adaptive Dark exAMples. Different from previous works that only rely on one positive and hard negatives as candidate passages, we create dark examples that all have moderate relevance to the query through mixing-up and masking in discrete space. Furthermore, as the quality of knowledge held in different training instances varies as measured by the teacher's confidence score, we propose a self-paced distillation strategy that adaptively concentrates on a subset of high-quality instances to conduct our dark-example-based knowledge distillation to help the student learn better. We conduct experiments on two widely-used benchmarks and verify the effectiveness of our method.

We present Hybrid Infused Reranking for Passages Retrieval (HYRR), a framework for training rerankers based on a hybrid of BM25 and neural retrieval models. Retrievers based on hybrid models have been shown to outperform both BM25 and neural models alone. Our approach exploits this improved performance when training a reranker, leading to a robust reranking model. The reranker, a cross-attention neural model, is shown to be robust to different first-stage retrieval systems, achieving better performance than rerankers simply trained upon the first-stage retrievers in the multi-stage systems. We present evaluations on a supervised passage retrieval task using MS MARCO and zero-shot retrieval tasks using BEIR. The empirical results show strong performance on both evaluations.

可区分的搜索索引（DSI）是一个新的新兴范式，用于信息检索。与索引和检索是两个不同且独立的组件的传统检索体系结构不同，DSI使用单个变压器模型执行索引和检索。在本文中，我们确定并解决了当前DSI模型的重要问题：DSI索引和检索过程之间发生的数据分布不匹配。具体而言，我们认为，在索引时，当前的DSI方法学会学会在长文档文本及其标识之间建立连接，但是在检索中，向DSI模型提供了简短的查询文本以执行文档标识符的检索。当使用DSI进行跨语言检索时，此问题进一步加剧，其中文档文本和查询文本使用不同的语言。为了解决当前DSI模型的这个基本问题，我们为DSI称为DSI-QG的简单而有效的索引框架。在DSI-QG中，文档由索引时间的查询生成模型生成的许多相关查询表示。这允许DSI模型在索引时将文档标识符连接到一组查询文本，因此减轻索引和检索阶段之间存在的数据分布不匹配。流行的单语言和跨语性通过基准数据集的经验结果表明，DSI-QG明显优于原始DSI模型。

Long document retrieval aims to fetch query-relevant documents from a large-scale collection, where knowledge distillation has become de facto to improve a retriever by mimicking a heterogeneous yet powerful cross-encoder. However, in contrast to passages or sentences, retrieval on long documents suffers from the scope hypothesis that a long document may cover multiple topics. This maximizes their structure heterogeneity and poses a granular-mismatch issue, leading to an inferior distillation efficacy. In this work, we propose a new learning framework, fine-grained distillation (FGD), for long-document retrievers. While preserving the conventional dense retrieval paradigm, it first produces global-consistent representations crossing different fine granularity and then applies multi-granular aligned distillation merely during training. In experiments, we evaluate our framework on two long-document retrieval benchmarks, which show state-of-the-art performance.

近年来，在应用预训练的语言模型（例如Bert）上，取得了巨大进展，以获取信息检索（IR）任务。在网页中通常使用的超链接已被利用用于设计预训练目标。例如，超链接的锚文本已用于模拟查询，从而构建了巨大的查询文档对以进行预训练。但是，作为跨越两个网页的桥梁，尚未完全探索超链接的潜力。在这项工作中，我们专注于建模通过超链接连接的两个文档之间的关系，并为临时检索设计一个新的预训练目标。具体而言，我们将文档之间的关系分为四组：无链接，单向链接，对称链接和最相关的对称链接。通过比较从相邻组采样的两个文档，该模型可以逐渐提高其捕获匹配信号的能力。我们提出了一个渐进的超链接预测（{php}）框架，以探索预训练中超链接的利用。对两个大规模临时检索数据集和六个提问数据集的实验结果证明了其优于现有的预训练方法。

多跳的推理（即跨两个或多个文档的推理）是NLP模型的关键要素，该模型利用大型语料库表现出广泛的知识。为了检索证据段落，多跳模型必须与整个啤酒花的快速增长的搜索空间抗衡，代表结合多个信息需求的复杂查询，并解决有关在训练段落之间跳出的最佳顺序的歧义。我们通过Baleen解决了这些问题，Baleen可以提高多跳检索的准确性，同时从多跳的训练信号中学习强大的训练信号的准确性。为了驯服搜索空间，我们提出了凝结的检索，该管道总结了每个跃点后检索到单个紧凑型上下文的管道。为了建模复杂的查询，我们引入了一个重点的后期相互作用检索器，该检索器允许同一查询表示的不同部分匹配不同的相关段落。最后，为了推断无序的训练段落中的跳跃依赖性，我们设计了潜在的跳跃订购，这是一种弱者的策略，在该策略中，受过训练的检索员本身选择了啤酒花的顺序。我们在检索中评估Baleen的两跳问答和多跳的要求验证，并确定最先进的绩效。

Open-domain question answering relies on efficient passage retrieval to select candidate contexts, where traditional sparse vector space models, such as TF-IDF or BM25, are the de facto method. In this work, we show that retrieval can be practically implemented using dense representations alone, where embeddings are learned from a small number of questions and passages by a simple dualencoder framework. When evaluated on a wide range of open-domain QA datasets, our dense retriever outperforms a strong Lucene-BM25 system greatly by 9%-19% absolute in terms of top-20 passage retrieval accuracy, and helps our end-to-end QA system establish new state-of-the-art on multiple open-domain QA benchmarks. 1 * Equal contribution 1 The code and trained models have been released at https://github.com/facebookresearch/DPR.

Recent developments of dense retrieval rely on quality representations of queries and contexts coming from pre-trained query and context encoders. In this paper, we introduce TouR (test-time optimization of query representations), which further optimizes instance-level query representations guided by signals from test-time retrieval results. We leverage a cross-encoder re-ranker to provide fine-grained pseudo labels over retrieval results and iteratively optimize query representations with the gradient descent method. Our theoretical analysis reveals that TouR can be viewed as a generalization of the classical Rocchio's algorithm for pseudo relevance feedback, and we present two variants leveraging psuedo labels as either hard binary or soft continuous labels. We first apply TouR on phrase retrieval with our proposed phrase re-ranker. On passage retrieval, we demonstrate its effectiveness with an off-the-shelf re-ranker. TouR improves the end-to-end open-domain QA accuracy significantly, as well as passage retrieval performance. Compared to re-ranker, TouR requires a smaller number of candidates, and achieves consistently better performance and runs up to 4x faster with our efficient implementation.

知识蒸馏是将知识从强大的教师转移到有效的学生模型的有效方法。理想情况下，我们希望老师越好，学生越好。但是，这种期望并不总是成真。通常，由于教师和学生之间的不可忽略的差距，更好的教师模型通过蒸馏导致不良学生。为了弥合差距，我们提出了一种渐进式蒸馏方法，以进行致密检索。产品由教师渐进式蒸馏和数据进行渐进的蒸馏组成，以逐步改善学生。我们对五个广泛使用的基准，MARCO通道，TREC Passage 19，TREC文档19，MARCO文档和自然问题进行了广泛的实验，其中POD在蒸馏方法中实现了密集检索的最新方法。代码和模型将发布。

神经信息检索（IR）具有极大的搜索和其他知识密集型语言任务。虽然许多神经IR方法将查询和文档编码为单载表示，但后期交互模型在每个令牌的粒度下产生多向量表示，并将相关性建模分解为可伸缩的令牌级计算。这种分解已被证明可以使迟到的交互更有效，但它以幅度的数量级膨胀这些模型的空间占地面积。在这项工作中，我们介绍了Colbertv2，这是一种猎犬，其与去噪的监督策略相结合的侵略性的残余压缩机制，同时提高晚期互动的质量和空间足迹。我们在各种基准中评估COLBertv2，在培训域内和外部建立最先进的质量，同时减少了晚期互动模型的空间足迹5-8 $ \ times $。

搜索会话中的上下文信息对于捕获用户的搜索意图很重要。已经提出了各种方法来对用户行为序列进行建模，以改善会话中的文档排名。通常，（搜索上下文，文档）对的训练样本在每个训练时期随机采样。实际上，了解用户的搜索意图和判断文档的相关性的困难从一个搜索上下文到另一个搜索上下文有很大差异。混合不同困难的训练样本可能会使模型的优化过程感到困惑。在这项工作中，我们为上下文感知文档排名提出了一个课程学习框架，其中排名模型以易于恐惧的方式学习搜索上下文和候选文档之间的匹配信号。这样一来，我们旨在将模型逐渐指向全球最佳。为了利用正面和负面示例，设计了两个课程。两个真实查询日志数据集的实验表明，我们提出的框架可以显着提高几种现有方法的性能，从而证明课程学习对上下文感知文档排名的有效性。

关于信息检索的许多最新研究集中在如何从一项任务（通常具有丰富的监督数据）转移到有限的其他各种任务，并隐含地假设可以从一个任务概括到所有其余的任务。但是，这忽略了这样一个事实，即有许多多样化和独特的检索任务，每个任务都针对不同的搜索意图，查询和搜索域。在本文中，我们建议使用几乎没有散热的检索，每个任务都有一个简短的描述和一些示例。为了扩大一些示例的功能，我们提出了针对检索器（即将到来）的及时基本查询生成，该查询将大型语言模型（LLM）作为几个弹片查询生成器，并根据生成的数据创建特定于任务的检索器。通过LLM的概括能力提供动力，即要来源使得可以仅基于一些示例{没有自然问题或MS MARCO来训练％问题生成器或双重编码器，就可以仅基于一些示例{没有}来创建特定于任务的端到端检索。出乎意料的是，LLM提示不超过8个示例，允许双重编码器在MARCO（例如Colbert V2）上训练的大量工程模型平均在11个检索套件中超过1.2 NDCG。使用相同生成数据的进一步培训标准尺寸的重新级别可获得5.0点NDCG的改进。我们的研究确定，查询产生比以前观察到的更有效，尤其是在给出少量特定于任务知识的情况下。

Recent progress in Natural Language Understanding (NLU) is driving fast-paced advances in Information Retrieval (IR), largely owed to ne-tuning deep language models (LMs) for document ranking.While remarkably e ective, the ranking models based on these LMs increase computational cost by orders of magnitude over prior approaches, particularly as they must feed each query-document pair through a massive neural network to compute a single relevance score. To tackle this, we present ColBERT, a novel ranking model that adapts deep LMs (in particular, BERT) for e cient retrieval. ColBERT introduces a late interaction architecture that independently encodes the query and the document using BERT and then employs a cheap yet powerful interaction step that models their ne-grained similarity. By delaying and yet retaining this negranular interaction, ColBERT can leverage the expressiveness of deep LMs while simultaneously gaining the ability to pre-compute document representations o ine, considerably speeding up query processing. Beyond reducing the cost of re-ranking the documents retrieved by a traditional model, ColBERT's pruning-friendly interaction mechanism enables leveraging vector-similarity indexes for end-to-end retrieval directly from a large document collection. We extensively evaluate ColBERT using two recent passage search datasets. Results show that ColBERT's e ectiveness is competitive with existing BERT-based models (and outperforms every non-BERT baseline), while executing two orders-of-magnitude faster and requiring four orders-of-magnitude fewer FLOPs per query.

排名者在事实上的“检索和rerank”管道中起着必不可少的作用，但其训练仍然落后 - 从中​​度的负面因素或/和/和/和作为回收者的辅助模块中学习。在这项工作中，我们首先确定了强大的排名者的两个主要障碍，即是由训练有素的回猎犬和非理想的负面负面的固有标签噪声，该噪声是为高能力的排名所采样的。因此，我们提出多个检索器，因为负面发电机改善了排名者的鲁棒性，其中i）涉及广泛的分发标签噪声，使排名者与每个噪声分布相对，而ii）与排名相对较接近排名负分配，导致更具挑战性的培训。为了评估我们的强大排名者（称为r $^2 $ anker），我们在各种环境中进行了有关流行通道检索基准测试的各种实验，包括BM25级，全等级，回收者蒸馏等。经验结果验证了新的州 - 新州 - 新州 - 我们模型的效果。