近年来，大型语言模型（LLMS）在自然语言产生中表现出了令人印象深刻的实力。提高发电多样性的一种常见做法是从模型中采样多个输出。但是，缺乏一种简单且可靠的方式来从这些随机样品中选择最佳输出。作为一个案例研究，在问题产生的背景下，我们提出了两种基于迅速的方法，以从一组LLM生成的候选人中选择高质量问题。我们的方法在1）限制下起作用，一个黑框（不可修改）问题生成模型和2）缺乏访问人类宣传的参考文献 - 这两者都是现实世界中LLMS的现实局限性。通过自动和人类评估，我们从经验上证明，我们的方法可以有效地选择比贪婪的生成更高质量的问题。

Powerful generative models have led to recent progress in question generation (QG). However, it is difficult to measure advances in QG research since there are no standardized resources that allow a uniform comparison among approaches. In this paper, we introduce QG-Bench, a multilingual and multidomain benchmark for QG that unifies existing question answering datasets by converting them to a standard QG setting. It includes general-purpose datasets such as SQuAD for English, datasets from ten domains and two styles, as well as datasets in eight different languages. Using QG-Bench as a reference, we perform an extensive analysis of the capabilities of language models for the task. First, we propose robust QG baselines based on fine-tuning generative language models. Then, we complement automatic evaluation based on standard metrics with an extensive manual evaluation, which in turn sheds light on the difficulty of evaluating QG models. Finally, we analyse both the domain adaptability of these models as well as the effectiveness of multilingual models in languages other than English. QG-Bench is released along with the fine-tuned models presented in the paper https://github.com/asahi417/lm-question-generation, which are also available as a demo https://autoqg.net/.

大型语言模型越来越能够通过相对较少的特定任务的监督产生流畅的出现文本。但这些模型可以准确解释分类决策吗？我们考虑使用少量人写的例子（即，以几滴方式）生成自由文本解释的任务。我们发现（1）创作更高质量的例子，以提示导致更高质量的世代; （2）令人惊讶的是，在头到头比较中，人群公司通常更喜欢GPT-3生成的解释，以众包中包含的人性写入的解释。然而，Crowdworker评级也表明，虽然模型产生了事实，语法和充分的解释，但它们具有改进的空间，例如沿着提供新颖信息和支持标签的轴。我们创建了一种管道，该管道将GPT-3与监督过滤器结合起来，该过滤器通过二进制可接受性判断来包含人类循环。尽管具有重要的主观性内在的判断可接受性，但我们的方法能够始终如一地过滤人类可接受的GPT-3生成的解释。

本文探讨了提高语言模型的零次学习能力的简单方法。我们表明，指令调整 - 通过对说明书中所述的任务集合微调语言模型 - 大幅提升零射门上看不见任务中的表现。我们采取预训练的语言模型和指令调整它通过自然语言指令模板语言表达了60NLP任务137B参数。我们评估这种指令调整模型，我们称之为FLAN，在看不见的任务类型。FLAN显着改善其未修饰的对应的性能和超过25的20个任务，我们评估零射门175BGPT-3。FLAN甚至GPT-3通过在安利，RTE，BoolQ，AI2-ARC，OpenbookQA和StoryCloze大比分胜过几拍。消融研究显示任务和模型的规模，这个数字是指令调整取得成功的关键组成部分。

Existing metrics for evaluating the quality of automatically generated questions such as BLEU, ROUGE, BERTScore, and BLEURT compare the reference and predicted questions, providing a high score when there is a considerable lexical overlap or semantic similarity between the candidate and the reference questions. This approach has two major shortcomings. First, we need expensive human-provided reference questions. Second, it penalises valid questions that may not have high lexical or semantic similarity to the reference questions. In this paper, we propose a new metric, RQUGE, based on the answerability of the candidate question given the context. The metric consists of a question-answering and a span scorer module, in which we use pre-trained models from the existing literature, and therefore, our metric can be used without further training. We show that RQUGE has a higher correlation with human judgment without relying on the reference question. RQUGE is shown to be significantly more robust to several adversarial corruptions. Additionally, we illustrate that we can significantly improve the performance of QA models on out-of-domain datasets by fine-tuning on the synthetic data generated by a question generation model and re-ranked by RQUGE.

知识密集型任务，例如开放域问题答案（QA），需要访问大量的世界知识或领域知识。知识密集型任务的一种常见方法是采用检索到阅读的管道，该管道首先从诸如Wikipedia之类的外部语料库中检索少数相关的上下文文档，然后预测在检索文档的条件下得到答案。在本文中，我们提出了一种新的观点，可以通过用大型语言模型生成器代替文档检索器来解决知识密集型任务。我们称我们的方法生成-Read Read（GenRead），该方法首先提示大型语言模型根据给定问题生成上下文文档，然后读取生成的文档以产生最终答案。此外，我们提出了一种基于聚类的提示方法，该方法选择了不同的提示，从而产生了涵盖不同观点的生成文档，从而更好地回忆了可接受的答案。我们对三个不同的知识密集任务进行了广泛的实验，包括开放域质量检查，事实检查和对话系统。值得注意的是，GenRead在Triviaqa和WebQ上实现了71.6和54.4的精确匹配分数，显着超过了最先进的检索到+4.0和+3.9的最先进的dpr-fid，而无需从任何外部知识源中检索任何文档。最后，我们证明可以通过结合检索和生成来进一步提高模型性能。

Human evaluation is the foundation upon which the evaluation of both summarization systems and automatic metrics rests. However, existing human evaluation protocols and benchmarks for summarization either exhibit low inter-annotator agreement or lack the scale needed to draw statistically significant conclusions, and an in-depth analysis of human evaluation is lacking. In this work, we address the shortcomings of existing summarization evaluation along the following axes: 1) We propose a modified summarization salience protocol, Atomic Content Units (ACUs), which relies on fine-grained semantic units and allows for high inter-annotator agreement. 2) We curate the Robust Summarization Evaluation (RoSE) benchmark, a large human evaluation dataset consisting of over 22k summary-level annotations over state-of-the-art systems on three datasets. 3) We compare our ACU protocol with three other human evaluation protocols, underscoring potential confounding factors in evaluation setups. 4) We evaluate existing automatic metrics using the collected human annotations across evaluation protocols and demonstrate how our benchmark leads to more statistically stable and significant results. Furthermore, our findings have important implications for evaluating large language models (LLMs), as we show that LLMs adjusted by human feedback (e.g., GPT-3.5) may overfit unconstrained human evaluation, which is affected by the annotators' prior, input-agnostic preferences, calling for more robust, targeted evaluation methods.

Explaining the black-box predictions of NLP models naturally and accurately is an important open problem in natural language generation. These free-text explanations are expected to contain sufficient and carefully-selected evidence to form supportive arguments for predictions. Due to the superior generative capacity of large pretrained language models, recent work built on prompt engineering enables explanation generation without specific training. However, explanation generated through single-pass prompting often lacks sufficiency and conciseness. To address this problem, we develop an information bottleneck method EIB to produce refined explanations that are sufficient and concise. Our approach regenerates the free-text explanation by polishing the single-pass output from the pretrained language model but retaining the information that supports the contents being explained. Experiments on two out-of-domain tasks verify the effectiveness of EIB through automatic evaluation and thoroughly-conducted human evaluation.

Large language models show improved downstream task performance when prompted to generate step-by-step reasoning to justify their final answers. These reasoning steps greatly improve model interpretability and verification, but objectively studying their correctness (independent of the final answer) is difficult without reliable methods for automatic evaluation. We simply do not know how often the stated reasoning steps actually support the final end task predictions. In this work, we present ROSCOE, a suite of interpretable, unsupervised automatic scores that improve and extend previous text generation evaluation metrics. To evaluate ROSCOE against baseline metrics, we design a typology of reasoning errors and collect synthetic and human evaluation scores on commonly used reasoning datasets. In contrast with existing metrics, ROSCOE can measure semantic consistency, logicality, informativeness, fluency, and factuality - among other traits - by leveraging properties of step-by-step rationales. We empirically verify the strength of our metrics on five human annotated and six programmatically perturbed diagnostics datasets - covering a diverse set of tasks that require reasoning skills and show that ROSCOE can consistently outperform baseline metrics.

迄今为止对文本生成的评估主要集中在依次创建的内容上，而不是对文本的改进。但是，写作自然是一个迭代和增量过程，需要在不同的模块化技能上进行专业知识，例如修复过时的信息或使样式更加一致。即便如此，对模型执行这些技能和编辑能力的模型能力的全面评估仍然很少。这项工作介绍了EditeVal：基于指导的，基准和评估套件，该套件利用现有的现有和新数据集自动评估编辑功能，例如使文本更具凝聚力和释义。我们评估了几种预训练的模型，这表明指令和同伴表现最好，但是大多数基准都落在监督的SOTA以下，尤其是在中和和更新信息时。我们的分析还表明，用于编辑任务的常用指标并不总是很好地关联，并且对具有最高性能的提示的优化并不一定带来对不同模型的最强鲁棒性。通过发布此基准和公开可用的排行榜挑战，我们希望在开发能够迭代和更可控制的编辑模型中解锁未来的研究。

从预训练的语言模型中进行的引导已被证明是用于建立基础视觉模型（VLM）的有效方法，例如图像字幕或视觉问题的答案。但是，很难用它来使模型符合用户的理由来获得特定答案。为了引起和加强常识性原因，我们提出了一个迭代采样和调整范式，称为Illume，执行以下循环：给定图像问题提示提示，VLM采样了多个候选人，并通过人类评论家通过偏好提供最小的反馈。选择，用于微调。该循环增加了训练数据，并逐渐雕刻出VLM的合理化功能。我们的详尽实验表明，Illume在使用较少的培训数据的同时，仅需要最少的反馈，与标准监督的微调竞争。

This paper proposes a question-answering system that can answer questions whose supporting evidence is spread over multiple (potentially long) documents. The system, called Visconde, uses a three-step pipeline to perform the task: decompose, retrieve, and aggregate. The first step decomposes the question into simpler questions using a few-shot large language model (LLM). Then, a state-of-the-art search engine is used to retrieve candidate passages from a large collection for each decomposed question. In the final step, we use the LLM in a few-shot setting to aggregate the contents of the passages into the final answer. The system is evaluated on three datasets: IIRC, Qasper, and StrategyQA. Results suggest that current retrievers are the main bottleneck and that readers are already performing at the human level as long as relevant passages are provided. The system is also shown to be more effective when the model is induced to give explanations before answering a question. Code is available at \url{https://github.com/neuralmind-ai/visconde}.

在有问题的回答需要常识的问题上，语言模型（例如，GPT-3）已用于生成表达有助于提高性能的背景知识的文本。然而，使用此类模型的成本很高。在这项工作中，我们对较小的语言模型产生有用的中间上下文，此处称为阐述。我们的框架在更新两个语言模型之间交替使用 - 阐述生成器和一个答案预测变量 - 允许每个语言都影响彼此。我们的模型使用少于GPT-3的参数的0.5％优于具有相似尺寸的替代方案，并在四个常识性问题上回答基准测试的GPT-3上的差距缩小。人类评估表明，生成的阐述的质量很高。

Instruction tuning enables pretrained language models to perform new tasks from inference-time natural language descriptions. These approaches rely on vast amounts of human supervision in the form of crowdsourced datasets or user interactions. In this work, we introduce Unnatural Instructions: a large dataset of creative and diverse instructions, collected with virtually no human labor. We collect 64,000 examples by prompting a language model with three seed examples of instructions and eliciting a fourth. This set is then expanded by prompting the model to rephrase each instruction, creating a total of approximately 240,000 examples of instructions, inputs, and outputs. Experiments show that despite containing a fair amount of noise, training on Unnatural Instructions rivals the effectiveness of training on open-source manually-curated datasets, surpassing the performance of models such as T0++ and Tk-Instruct across various benchmarks. These results demonstrate the potential of model-generated data as a cost-effective alternative to crowdsourcing for dataset expansion and diversification.

Large pre-trained language models have recently enabled open-ended generation frameworks (e.g., prompt-to-text NLG) to tackle a variety of tasks going beyond the traditional data-to-text generation. While this framework is more general, it is under-specified and often leads to a lack of controllability restricting their real-world usage. We propose a new grounded keys-to-text generation task: the task is to generate a factual description about an entity given a set of guiding keys, and grounding passages. To address this task, we introduce a new dataset, called EntDeGen. Inspired by recent QA-based evaluation measures, we propose an automatic metric, MAFE, for factual correctness of generated descriptions. Our EntDescriptor model is equipped with strong rankers to fetch helpful passages and generate entity descriptions. Experimental result shows a good correlation (60.14) between our proposed metric and human judgments of factuality. Our rankers significantly improved the factual correctness of generated descriptions (15.95% and 34.51% relative gains in recall and precision). Finally, our ablation study highlights the benefit of combining keys and groundings.

预测任务标签和为其预测生成自由文本阐述的自律化模型可以实现与NLP系统更直观的交互。然而，这些模型目前正在接受大量人为的自由文本解释，每个任务都会阻碍更广泛的使用。我们建议使用少数培训例子研究更现实的自律化建立。我们出示2月 - 一个标准化的四个现有英语数据集和相关指标。我们通过2月份广泛探索自然语言提示来确定正确的提示方法。然后，通过使用此提示并缩放模型大小，我们证明了几次拍摄自合合理化的进展。我们展示了这项任务的完善房间仍然有充足的改进空间：人类注册人评估的生成解释的平均合理性最多为51％，而人类解释的合理性是76％。我们希望2月份与我们的拟议方法一起促使社区承担几次拍摄的自我合理化挑战。

Naturally-occurring information-seeking questions often contain questionable assumptions -- assumptions that are false or unverifiable. Questions containing questionable assumptions are challenging because they require a distinct answer strategy that deviates from typical answers to information-seeking questions. For instance, the question "When did Marie Curie discover Uranium?" cannot be answered as a typical when question without addressing the false assumption "Marie Curie discovered Uranium". In this work, we propose (QA)$^2$ (Question Answering with Questionable Assumptions), an open-domain evaluation dataset consisting of naturally-occurring search engine queries that may or may not contain questionable assumptions. To be successful on (QA)$^2$, systems must be able to detect questionable assumptions and also be able to produce adequate responses for both typical information-seeking questions and ones with questionable assumptions. We find that current models do struggle with handling questionable assumptions -- the best performing model achieves 59% human rater acceptability on abstractive QA with (QA)$^2$ questions, leaving substantial headroom for progress.

大型语言模型在零拍设置中显示出令人鼓舞的结果（Brown等，2020； Radford等，2019）。例如，他们只需在问题上调节并以最高概率选择答案来执行多项选择任务。但是，由于表面竞争的表面形式 - 在不同的表面形式竞争概率质量，即使它们代表相同的基本概念，例如“计算机”和“ PC”。由于概率质量是有限的，因此由于其他是有效答案的字符串的竞争（但不是多项选择选项之一），这会降低正确答案的概率。我们引入域有条件地互相信息，这是一种替代评分函数，可以通过简单地根据特定的零击任务的上下文中的先验可能性重新重新拨出每个选项来直接补偿表面竞争。在校准（Zhao等，2021）和所有GPT-2和GPT-3模型上，在各种多项选择数据集上，它都可以在零击性能方面的一致增长和未校准的评分功能。

最近已被证明大型语言模型在各种任务集中获得合理的零射普通化（Brown等，2020）。它已经假设这是语言模型的隐式多任务学习的结果，在语言模型中的预押（Radford等，2019）。可以通过明确的多任务学习直接引起零拍常规化？为了以缩放测试这个问题，我们开发一个系统，以便轻松地将任何自然语言任务映射到人类可读的提示表单中。我们转换一组大量的监督数据集，每个数据集都有多个提示，具有不同的措辞。这些提示的数据集允许基准测试模型执行完全看不见的任务的能力。我们介绍了一个普拉克尔编码器 - 解码器模型（Raffel等，2020; Lester等，2021），覆盖各种任务。该模型在多个标准数据集中达到强大的零点性能，通常优于其尺寸的型号超过16倍。此外，我们的方法对来自Big-替补基准测试的任务子集具有强烈性能，优于其尺寸的6倍。所有提示和培训的型号都可以在https://github.com/ bigscience-workshop / protectsource / httpsource / https：//huggingface.co/bigscience/t0pp。

Recent work has shown that fine-tuning large pre-trained language models on a collection of tasks described via instructions, a.k.a. instruction-tuning, improves their zero and few-shot generalization to unseen tasks. However, there is a limited understanding of the performance trade-offs of different decisions made during the instruction-tuning process. These decisions include the scale and diversity of the instruction-tuning benchmark, different task sampling strategies, fine-tuning with and without demonstrations, training using specialized datasets for reasoning and dialogue, and finally, the fine-tuning objectives themselves. In this paper, we characterize the effect of instruction-tuning decisions on downstream task performance when scaling both model and benchmark sizes. To this end, we create OPT-IML Bench: a large benchmark for Instruction Meta-Learning (IML) of 2000 NLP tasks consolidated into task categories from 8 existing benchmarks, and prepare an evaluation framework to measure three types of model generalizations: to tasks from fully held-out categories, to held-out tasks from seen categories, and to held-out instances from seen tasks. Through the lens of this framework, we first present insights about instruction-tuning decisions as applied to OPT-30B and further exploit these insights to train OPT-IML 30B and 175B, which are instruction-tuned versions of OPT. OPT-IML demonstrates all three generalization abilities at both scales on four different evaluation benchmarks with diverse tasks and input formats -- PromptSource, FLAN, Super-NaturalInstructions, and UnifiedSKG. Not only does it significantly outperform OPT on all benchmarks but is also highly competitive with existing models fine-tuned on each specific benchmark. We release OPT-IML at both scales, together with the OPT-IML Bench evaluation framework.