To facilitate research on text generation, this paper presents a comprehensive and unified library, TextBox 2.0, focusing on the use of pre-trained language models (PLMs). To be comprehensive, our library covers $13$ common text generation tasks and their corresponding $83$ datasets and further incorporates $45$ PLMs covering general, translation, Chinese, dialogue, controllable, distilled, prompting, and lightweight PLMs. We also implement $4$ efficient training strategies and provide $4$ generation objectives for pre-training new PLMs from scratch. To be unified, we design the interfaces to support the entire research pipeline (from data loading to training and evaluation), ensuring that each step can be fulfilled in a unified way. Despite the rich functionality, it is easy to use our library, either through the friendly Python API or command line. To validate the effectiveness of our library, we conduct extensive experiments and exemplify four types of research scenarios. The project is released at the link: https://github.com/RUCAIBox/TextBox.

预训练的语言模型（PLM）在自然语言生成（NLG）任务中取得了显着的成功。到目前为止，大多数PLM都使用大型一般语料库以无监督的方式进行了预培训。同时，与无监督的模型相比，预先训练的模型越来越多地显示出较低的数据表现出色。受监督预训练的成功的激励，我们提出了自然语言生成的多任务监督预训练（MVP）。为了预先培训文本生成模型MVP，我们从七个生成任务中收集了45个数据集的标记预训练语料库。对于每个任务，我们进一步预先训练特定的软提示，以刺激执行特定任务的模型能力。广泛的实验证明了我们在许多NLG任务中有监督的预训练的有效性，并且我们的一般方法在17个数据集中的12个中实现了最先进的性能。

Controllable Text Generation (CTG) is emerging area in the field of natural language generation (NLG). It is regarded as crucial for the development of advanced text generation technologies that are more natural and better meet the specific constraints in practical applications. In recent years, methods using large-scale pre-trained language models (PLMs), in particular the widely used transformer-based PLMs, have become a new paradigm of NLG, allowing generation of more diverse and fluent text. However, due to the lower level of interpretability of deep neural networks, the controllability of these methods need to be guaranteed. To this end, controllable text generation using transformer-based PLMs has become a rapidly growing yet challenging new research hotspot. A diverse range of approaches have emerged in the recent 3-4 years, targeting different CTG tasks which may require different types of controlled constraints. In this paper, we present a systematic critical review on the common tasks, main approaches and evaluation methods in this area. Finally, we discuss the challenges that the field is facing, and put forward various promising future directions. To the best of our knowledge, this is the first survey paper to summarize CTG techniques from the perspective of PLMs. We hope it can help researchers in related fields to quickly track the academic frontier, providing them with a landscape of the area and a roadmap for future research.

Neural models that do not rely on pre-training have excelled in the keyphrase generation task with large annotated datasets. Meanwhile, new approaches have incorporated pre-trained language models (PLMs) for their data efficiency. However, there lacks a systematic study of how the two types of approaches compare and how different design choices can affect the performance of PLM-based models. To fill in this knowledge gap and facilitate a more informed use of PLMs for keyphrase extraction and keyphrase generation, we present an in-depth empirical study. Formulating keyphrase extraction as sequence labeling and keyphrase generation as sequence-to-sequence generation, we perform extensive experiments in three domains. After showing that PLMs have competitive high-resource performance and state-of-the-art low-resource performance, we investigate important design choices including in-domain PLMs, PLMs with different pre-training objectives, using PLMs with a parameter budget, and different formulations for present keyphrases. Further results show that (1) in-domain BERT-like PLMs can be used to build strong and data-efficient keyphrase generation models; (2) with a fixed parameter budget, prioritizing model depth over width and allocating more layers in the encoder leads to better encoder-decoder models; and (3) introducing four in-domain PLMs, we achieve a competitive performance in the news domain and the state-of-the-art performance in the scientific domain.

如今，基础模型已成为人工智能中的基本基础设施之一，铺平了通往通用情报的方式。但是，现实提出了两个紧急挑战：现有的基础模型由英语社区主导；用户通常会获得有限的资源，因此不能总是使用基础模型。为了支持中文社区的发展，我们介绍了一个名为Fengshenbang的开源项目，该项目由认知计算与自然语言研究中心（CCNL）领导。我们的项目具有全面的功能，包括大型预培训模型，用户友好的API，基准，数据集等。我们将所有这些都包装在三个子项目中：风水次模型，风水框架和狂热基准。 Fengshenbang的开源路线图旨在重新评估中国预培训的大型大型模型的开源社区，促使整个中国大型模型社区的发展。我们还希望构建一个以用户为中心的开源生态系统，以允许个人访问所需的模型以匹配其计算资源。此外，我们邀请公司，大学和研究机构与我们合作建立大型开源模型的生态系统。我们希望这个项目将成为中国认知情报的基础。

电子健康记录（EHR）是现代医疗系统的重要组成部分，影响医疗保健提供，运营和研究。尽管在EHR中进行了结构化信息，但非结构化的文本仍吸引了很多关注，并已成为一个令人兴奋的研究领域。最近的神经自然语言处理（NLP）方法的成功导致了处理非结构化临床笔记的新方向。在这项工作中，我们创建了一个用于临床文本的Python库，Ehrkit。该库包含两个主要部分：模拟III特定功能和任务特定功能。第一部分介绍了用于访问MIMIC-III NoteEvents数据的接口列表，包括基本搜索，信息检索和信息提取。第二部分集成了许多第三方库，用于多达12个删除NLP任务，例如命名实体识别，摘要，机器翻译等。

语言是人类交流的主要工具，其中幽默是最有吸引力的部分之一。使用计算机，又称自然语言生成（NLG）的人类产生自然语言，已广泛用于对话系统，聊天机器人，机器翻译以及计算机AID创建，例如Idea Generations，剧本。但是，自然语言的幽默方面相对不足，尤其是在预训练的语言模型时代。在这项工作中，我们旨在初步测试NLG是否可以像人类一样产生幽默。我们构建了一个新的数据集，该数据集由众多数字化的中国可笑的串扰脚本（称为c $^3 $简称），该脚本适用于1800年代以来名为“ Xiangsheng”的流行中国表演艺术。 （为了方便非中国扬声器，我们在本文中称为“ Xiangsheng”的“ Crosstalk”。）我们基准了各种一代方法，包括训练seq2seq，微调中级PLMS和大型PLMS（大型PLMS）（有无微调）。此外，我们还进行了人类评估，表明1）大规模预处理在很大程度上提高了串扰的产生质量； 2）即使是从最佳PLM产生的脚本也远非我们的期望，只有65％的人类创建的串扰质量。我们得出结论，使用大型PLM可以在很大程度上改善幽默的产生，但仍处于起步阶段。 \ url {https://github.com/anonno2/crosstalk-generation}公开可用数据和基准代码。

最近的工作表明，（1）增加输入长度或（2）增加模型大小可以提高基于变压器的神经模型的性能。在本文中，我们提出了一个名为Longt5的新模型，我们探讨了同时缩放输入长度和模型大小的效果。具体而言，我们综合了从长输入变压器（ETC）的关注思路，并采用了从摘要预训练（PEGASU）的预训练策略进入可扩展的T5架构。结果是我们称之为{\ EM瞬态全球}（TGLOBAL）的新关注机制，这些机制是模仿等本地/全球注意力机制，但不需要额外的侧面输入。我们能够实现最先进的结果，以若干摘要任务，优于问题应答任务的原始T5模型。

Recent progress in natural language processing has been driven by advances in both model architecture and model pretraining. Transformer architectures have facilitated building higher-capacity models and pretraining has made it possible to effectively utilize this capacity for a wide variety of tasks. Transformers is an open-source library with the goal of opening up these advances to the wider machine learning community. The library consists of carefully engineered stateof-the art Transformer architectures under a unified API. Backing this library is a curated collection of pretrained models made by and available for the community. Transformers is designed to be extensible by researchers, simple for practitioners, and fast and robust in industrial deployments. The library is available at https://github.com/ huggingface/transformers.

本文对过去二十年来对自然语言生成（NLG）的研究提供了全面的审查，特别是与数据到文本生成和文本到文本生成深度学习方法有关，以及NLG的新应用技术。该调查旨在（a）给出关于NLG核心任务的最新综合，以及该领域采用的建筑;（b）详细介绍各种NLG任务和数据集，并提请注意NLG评估中的挑战，专注于不同的评估方法及其关系;（c）强调一些未来的强调和相对近期的研究问题，因为NLG和其他人工智能领域的协同作用而增加，例如计算机视觉，文本和计算创造力。

Pre-trained language models have been successful in natural language generation (NLG) tasks. While various decoding methods have been employed, they often produce suboptimal results. We first present an empirical analysis of three NLG tasks: summarization, machine translation, and constrained text generation. We found that selecting the best output from the results of multiple decoding methods can significantly improve performance. To further improve reranking for NLG tasks, we proposed a novel method, \textsc{PairReranker}, which uses a single encoder and a pairwise loss function to jointly encode a source input and a pair of candidates and compare them. Experiments on three NLG tasks demonstrated the effectiveness and flexibility of \textsc{PairReranker}, showing strong results, compared with previous baselines. In addition, our \textsc{PairReranker} can generalize to significantly improve GPT-3 (text-davinci-003) results (e.g., 24.55\% on CommonGen and 11.35\% on WMT18 zh-en), even though our rerankers are not trained with any GPT-3 candidates.

在本文中，我们建议利用对话的独特特征，共享参与者的常识性知识，以解决总结它们的困难。我们提出了病态的框架，该框架使用常识推论作为其他背景。与以前仅依赖于输入对话的工作相比，Sick使用外部知识模型来生成丰富的常识推断，并选择具有基于相似性选择方法的最可能的推理。基于生病的，病人++的理解为监督，在总结多任务学习环境中的对话时，添加了产生常识推断的任务。实验结果表明，通过注入常识性知识，我们的框架比现有方法产生更多信息和一致的摘要。

Recent progress in natural language processing has been driven by advances in both model architecture and model pretraining. Transformer architectures have facilitated building higher-capacity models and pretraining has made it possible to effectively utilize this capacity for a wide variety of tasks. Transformers is an open-source library with the goal of opening up these advances to the wider machine learning community. The library consists of carefully engineered stateof-the art Transformer architectures under a unified API. Backing this library is a curated collection of pretrained models made by and available for the community. Transformers is designed to be extensible by researchers, simple for practitioners, and fast and robust in industrial deployments. The library is available at https://github.com/ huggingface/transformers.

我们提出了一项实证研究，以适应现有的经过验证的文本对文本模型，以备长期输入。通过沿预训练管道的三个轴的全面研究 - 模型架构，优化目标和训练式语料库，我们提出了一种有效的食谱，以从现有的短篇小说模型中构建长篇小说模型。具体而言，我们用汇总仪的块关注替换了变压器中的全部注意力，并使用蒙版的跨度预测任务为模型预算，长度不同。就训练训练的语料库而言，我们发现，与使用通常在其域覆盖范围中通常受到限制的现有长文档语料库相比，使用大型开放域语料库的随机串联的短篇小说可以提高性能。通过这些发现，我们建立了一个长篇文本模型，该模型可以在长篇文本质量检查任务上实现竞争性能，并在五个长文本摘要数据集上建立新的最新技术，通常优于先前的方法，具有较大的模型大小。

Stack Overflow是最受欢迎的编程社区之一，开发人员可以为他们遇到的问题寻求帮助。然而，如果没有经验的开发人员无法清楚地描述他们的问题，那么他们很难吸引足够的关注并获得预期的答案。我们提出了M $ _3 $ NSCT5，这是一种自动从给定代码片段生成多个帖子标题的新颖方法。开发人员可以使用生成的标题查找密切相关的帖子并完成其问题描述。 M $ _3 $ NSCT5使用Codet5骨干，这是一种具有出色语言理解和发电能力的预训练的变压器模型。为了减轻歧义问题，即在不同背景下可以将相同的代码片段与不同的标题保持一致，我们提出了最大的边缘多元核抽样策略，以一次产生多个高质量和不同的标题候选者，以便开发人员选择。我们构建了一个大规模数据集，其中包含890,000个问题帖子，其中涵盖了八种编程语言，以验证M $ _3 $ NSCT5的有效性。 BLEU和胭脂指标的自动评估结果表明，M $ _3 $ NSCT5的优势比六个最先进的基线模型。此外，具有值得信赖结果的人类评估也证明了我们对现实世界应用方法的巨大潜力。

Software engineers working with the same programming language (PL) may speak different natural languages (NLs) and vice versa, erecting huge barriers to communication and working efficiency. Recent studies have demonstrated the effectiveness of generative pre-training in computer programs, yet they are always English-centric. In this work, we step towards bridging the gap between multilingual NLs and multilingual PLs for large language models (LLMs). We release ERNIE-Code, a unified pre-trained language model for 116 NLs and 6 PLs. We employ two methods for universal cross-lingual pre-training: span-corruption language modeling that learns patterns from monolingual NL or PL; and pivot-based translation language modeling that relies on parallel data of many NLs and PLs. Extensive results show that ERNIE-Code outperforms previous multilingual LLMs for PL or NL across a wide range of end tasks of code intelligence, including multilingual code-to-text, text-to-code, code-to-code, and text-to-text generation. We further show its advantage of zero-shot prompting on multilingual code summarization and text-to-text translation. We will make our code and pre-trained models publicly available.

在本文中，我们利用了以前的预训练模型（PTM）的优势，并提出了一种新型的中国预训练的不平衡变压器（CPT）。与以前的中国PTM不同，CPT旨在利用自然语言理解（NLU）和自然语言生成（NLG）之间的共同知识来促进表现。 CPT包括三个部分：共享编码器，一个理解解码器和一代解码器。具有共享编码器的两个特定解码器分别通过蒙版语言建模（MLM）进行了预训练，并分别将自动编码（DAE）任务进行了验证。借助部分共享的体系结构和多任务预培训，CPT可以（1）使用两个解码器学习NLU或NLG任务的特定知识，并且（2）对模型的潜力充分利用了微调。此外，不平衡的变压器节省了计算和存储成本，这使CPT竞争激烈，并极大地加速了文本生成的推断。对各种中国NLU和NLG任务的实验结果显示了CPT的有效性。

Given a natural language that describes the user's demands, the NL2Code task aims to generate code that addresses the demands. This is a critical but challenging task that mirrors the capabilities of AI-powered programming. The NL2Code task is inherently versatile, diverse and complex. For example, a demand can be described in different languages, in different formats, and at different levels of granularity. This inspired us to do this survey for NL2Code. In this survey, we focus on how does neural network (NN) solves NL2Code. We first propose a comprehensive framework, which is able to cover all studies in this field. Then, we in-depth parse the existing studies into this framework. We create an online website to record the parsing results, which tracks existing and recent NL2Code progress. In addition, we summarize the current challenges of NL2Code as well as its future directions. We hope that this survey can foster the evolution of this field.

Pre-trained models have achieved remarkable success in natural language processing (NLP). However, existing pre-training methods underutilize the benefits of language understanding for generation. Inspired by the idea of Generative Adversarial Networks (GANs), we propose a GAN-style model for encoder-decoder pre-training by introducing an auxiliary discriminator, unifying the ability of language understanding and generation in a single model. Our model, named as GanLM, is trained with two pre-training objectives: replaced token detection and replaced token denoising. Specifically, given masked source sentences, the generator outputs the target distribution and the discriminator predicts whether the target sampled tokens from distribution are incorrect. The target sentence is replaced with misclassified tokens to construct noisy previous context, which is used to generate the gold sentence. In general, both tasks improve the ability of language understanding and generation by selectively using the denoising data. Extensive experiments in language generation benchmarks show that GanLM with the powerful language understanding capability outperforms various strong pre-trained language models (PLMs) and achieves state-of-the-art performance.

Natural Language Generation (NLG) has improved exponentially in recent years thanks to the development of sequence-to-sequence deep learning technologies such as Transformer-based language models. This advancement has led to more fluent and coherent NLG, leading to improved development in downstream tasks such as abstractive summarization, dialogue generation and data-to-text generation. However, it is also apparent that deep learning based generation is prone to hallucinate unintended text, which degrades the system performance and fails to meet user expectations in many real-world scenarios. To address this issue, many studies have been presented in measuring and mitigating hallucinated texts, but these have never been reviewed in a comprehensive manner before. In this survey, we thus provide a broad overview of the research progress and challenges in the hallucination problem in NLG. The survey is organized into two parts: (1) a general overview of metrics, mitigation methods, and future directions; and (2) an overview of task-specific research progress on hallucinations in the following downstream tasks, namely abstractive summarization, dialogue generation, generative question answering, data-to-text generation, machine translation, and visual-language generation. This survey serves to facilitate collaborative efforts among researchers in tackling the challenge of hallucinated texts in NLG.