在本文中，我们介绍了DOCMT5，这是一种预先培训的多语言序列到序列语言模型，具有大规模并行文档。虽然以前的方法专注于利用句子级并行数据，但我们尝试构建一个可以理解和生成长文件的通用预训练模型。我们提出了一个简单有效的预训练目标 - 文件重新排序机翻译（DRMT），其中需要翻译和屏蔽的输入文件。 DRMT在各种文档级生成任务中对强大基线带来一致的改进，包括超过12个BLEU积分，用于观看语言对文件级MT，超过7个BLEU积分，用于看不见的语言对文件级MT和3胭脂-1位为言语对交叉术概要。我们在WMT20 De-en和IWSLT15 Zh-ZH文档翻译任务中实现了最先进的（SOTA）。我们还对文档预培训的各种因素进行了广泛的分析，包括（1）预培训数据质量的影响和（2）组合单语言和交叉训练的影响。我们计划公开使用我们的模型检查站。

This paper demonstrates that multilingual denoising pre-training produces significant performance gains across a wide variety of machine translation (MT) tasks. We present mBART -a sequence-to-sequence denoising auto-encoder pre-trained on large-scale monolingual corpora in many languages using the BART objective . mBART is the first method for pre-training a complete sequence-to-sequence model by denoising full texts in multiple languages, while previous approaches have focused only on the encoder, decoder, or reconstructing parts of the text. Pre-training a complete model allows it to be directly fine tuned for supervised (both sentence-level and document-level) and unsupervised machine translation, with no task-specific modifications. We demonstrate that adding mBART initialization produces performance gains in all but the highest-resource settings, including up to 12 BLEU points for low resource MT and over 5 BLEU points for many document-level and unsupervised models. We also show it also enables new types of transfer to language pairs with no bi-text or that were not in the pre-training corpus, and present extensive analysis of which factors contribute the most to effective pre-training.

Despite the current success of multilingual pre-training, most prior works focus on leveraging monolingual data or bilingual parallel data and overlooked the value of trilingual parallel data. This paper presents \textbf{Tri}angular Document-level \textbf{P}re-training (\textbf{TRIP}), which is the first in the field to extend the conventional monolingual and bilingual pre-training to a trilingual setting by (i) \textbf{Grafting} the same documents in two languages into one mixed document, and (ii) predicting the remaining one language as the reference translation. Our experiments on document-level MT and cross-lingual abstractive summarization show that TRIP brings by up to 3.65 d-BLEU points and 6.2 ROUGE-L points on three multilingual document-level machine translation benchmarks and one cross-lingual abstractive summarization benchmark, including multiple strong state-of-the-art (SOTA) scores. In-depth analysis indicates that TRIP improves document-level machine translation and captures better document contexts in at least three characteristics: (i) tense consistency, (ii) noun consistency and (iii) conjunction presence.

Pre-trained models have achieved remarkable success in natural language processing (NLP). However, existing pre-training methods underutilize the benefits of language understanding for generation. Inspired by the idea of Generative Adversarial Networks (GANs), we propose a GAN-style model for encoder-decoder pre-training by introducing an auxiliary discriminator, unifying the ability of language understanding and generation in a single model. Our model, named as GanLM, is trained with two pre-training objectives: replaced token detection and replaced token denoising. Specifically, given masked source sentences, the generator outputs the target distribution and the discriminator predicts whether the target sampled tokens from distribution are incorrect. The target sentence is replaced with misclassified tokens to construct noisy previous context, which is used to generate the gold sentence. In general, both tasks improve the ability of language understanding and generation by selectively using the denoising data. Extensive experiments in language generation benchmarks show that GanLM with the powerful language understanding capability outperforms various strong pre-trained language models (PLMs) and achieves state-of-the-art performance.

Software engineers working with the same programming language (PL) may speak different natural languages (NLs) and vice versa, erecting huge barriers to communication and working efficiency. Recent studies have demonstrated the effectiveness of generative pre-training in computer programs, yet they are always English-centric. In this work, we step towards bridging the gap between multilingual NLs and multilingual PLs for large language models (LLMs). We release ERNIE-Code, a unified pre-trained language model for 116 NLs and 6 PLs. We employ two methods for universal cross-lingual pre-training: span-corruption language modeling that learns patterns from monolingual NL or PL; and pivot-based translation language modeling that relies on parallel data of many NLs and PLs. Extensive results show that ERNIE-Code outperforms previous multilingual LLMs for PL or NL across a wide range of end tasks of code intelligence, including multilingual code-to-text, text-to-code, code-to-code, and text-to-text generation. We further show its advantage of zero-shot prompting on multilingual code summarization and text-to-text translation. We will make our code and pre-trained models publicly available.

对于多语言序列到序列预审预周序模型（多语言SEQ2SEQ PLM），例如姆巴特（Mbart），自制的预处理任务接受了多种单语言的培训，例如25种来自CommonCrawl的语言，而下游的跨语言任务通常在双语语言子集上进行，例如英语 - 德国人，存在数据差异，即领域的差异，以及跨语言学习客观差异，即在训练和填充阶段之间的任务差异。为了弥合上述跨语言域和任务差距，我们将使用额外的代码切换恢复任务扩展了香草预后管道。具体而言，第一阶段采用自我监督的代码转换还原任务作为借口任务，从而允许多语言SEQ2SEQ PLM获取一些域内对齐信息。在第二阶段，我们正常在下游数据上微调模型。 NLG评估（12个双语翻译任务，30个零射击任务和2项跨语言摘要任务）和NLU评估（7个跨语性自然语言推理任务）的实验表明，我们的模型超过了强大的基线MBART，具有标准的FINETUNNING，这表明了我们的模型策略，一致。分析表明，我们的方法可以缩小跨语性句子表示的欧几里得距离，并通过微不足道的计算成本改善模型概括。我们在：https：//github.com/zanchangtong/csr4mbart上发布代码。

跨语性摘要是用一种语言（例如英语）以不同语言（例如中文）生成一种语言（例如英语）的摘要。在全球化背景下，这项任务吸引了计算语言学界的越来越多的关注。然而，对于这项任务仍然缺乏全面的审查。因此，我们在该领域的数据集，方法和挑战上介绍了第一个系统的批判性审查。具体而言，我们分别根据不同的构造方法和解决方案范例仔细组织现有的数据集和方法。对于每种类型的数据集或方法，我们彻底介绍并总结了以前的努力，并将它们相互比较以提供更深入的分析。最后，我们还讨论了有希望的方向，并提供了我们的思想，以促进未来的研究。这项调查适用于跨语性摘要的初学者和专家，我们希望它将成为起点，也可以为对该领域感兴趣的研究人员和工程师提供新的想法。

以前的工作主要侧重于改善NLU任务的交叉传输，具有多语言预用编码器（MPE），或提高与伯特的监督机器翻译的性能。然而，探索了，MPE是否可以有助于促进NMT模型的交叉传递性。在本文中，我们专注于NMT中的零射频转移任务。在此任务中，NMT模型培训，只有一个语言对的并行数据集和搁置架MPE，然后它直接测试在零拍语言对上。我们为此任务提出了Sixt，一个简单而有效的模型。 SIXT利用了两阶段培训计划利用MPE，并进一步改进了解离编码器和容量增强的解码器。使用此方法，SIMPT显着优于MBart，这是一个用于NMT的预磨削的多语言编码器解码器模型，平均改善了14个源语言的零拍摄的任何英语测试集上的7.1 BLEU。此外，培训计算成本和培训数据较少，我们的模型在15个任何英语测试组上实现了比Criss和M2M-100，两个强大的多语言NMT基线更好的性能。

Universal cross-lingual sentence embeddings map semantically similar cross-lingual sentences into a shared embedding space. Aligning cross-lingual sentence embeddings usually requires supervised cross-lingual parallel sentences. In this work, we propose mSimCSE, which extends SimCSE to multilingual settings and reveal that contrastive learning on English data can surprisingly learn high-quality universal cross-lingual sentence embeddings without any parallel data. In unsupervised and weakly supervised settings, mSimCSE significantly improves previous sentence embedding methods on cross-lingual retrieval and multilingual STS tasks. The performance of unsupervised mSimCSE is comparable to fully supervised methods in retrieving low-resource languages and multilingual STS. The performance can be further enhanced when cross-lingual NLI data is available. Our code is publicly available at https://github.com/yaushian/mSimCSE.

本文介绍了Z-Code ++，这是一种针对抽象文本摘要优化的新的预训练的语言模型。该模型使用三种技术扩展了艺术编码器模型的状态。首先，我们使用两阶段的预训练过程来改善模型在低资源摘要任务上的性能。该模型首先是使用文本语料库进行语言理解的预先培训的，然后在汇总语料库中不断预先培训，以进行基础文本生成。其次，我们用分离的注意力层代替编码器中的自我发项层，其中每个单词都使用两个向量分别代表其内容和位置。第三，我们使用融合编码器，这是一种以层次方式编码长序列的简单而有效的方法。 Z-Code ++在13个文本摘要任务中的9个跨5种语言中创建了新的艺术状态。我们的模型的参数有效，因为它的表现优于XSUM上600倍较大的Palm-540b，并且在Samsum上的易经的200倍GPT3-175B较大。在零射击和少量设置中，我们的模型大大优于竞争模型。

我们提出了一种两阶段的培训方法，用于开发单个NMT模型，以翻译英语和英语的看不见的语言。对于第一阶段，我们将编码器模型初始化以鉴定XLM-R和Roberta的权重，然后对25种语言的平行数据进行多种语言微调。我们发现该模型可以推广到对看不见的语言的零击翻译。在第二阶段，我们利用这种概括能力从单语数据集生成合成的并行数据，然后用连续的反向翻译训练。最终模型扩展到了英语到许多方向，同时保持了多到英语的性能。我们称我们的方法为ecxtra（以英语为中心的跨语言（x）转移）。我们的方法依次利用辅助并行数据和单语言数据，并且在概念上很简单，仅在两个阶段都使用标准的跨熵目标。最终的ECXTRA模型对8种低资源语言的无监督NMT进行了评估，该语言为英语至哈萨克语（22.3> 10.4 bleu）以及其他15个翻译方向的竞争性能而获得了新的最先进。

我们介绍了MTG，这是一套新的基准套件，用于培训和评估多语言文本生成。它是具有最大人类通知数据（400K）的第一次传播的多语言多路文本生成数据集。它包括五种语言（英语，德语，法语，西班牙语和中文）的四代任务（故事产生，问题生成，标题生成和文本摘要）。Multiway设置可以启用跨语言和任务的模型测试知识传输功能。使用MTG，我们从不同方面训练和分析了几种流行的多语言生成模型。我们的基准套件通过更多的人为宣传的并行数据促进了模型性能增强。它提供了各种一代方案的全面评估。代码和数据可在\ url {https://github.com/zide05/mtg}上获得。

State-of-the-art natural language processing systems rely on supervision in the form of annotated data to learn competent models. These models are generally trained on data in a single language (usually English), and cannot be directly used beyond that language. Since collecting data in every language is not realistic, there has been a growing interest in crosslingual language understanding (XLU) and low-resource cross-language transfer. In this work, we construct an evaluation set for XLU by extending the development and test sets of the Multi-Genre Natural Language Inference Corpus (MultiNLI) to 15 languages, including low-resource languages such as Swahili and Urdu. We hope that our dataset, dubbed XNLI, will catalyze research in cross-lingual sentence understanding by providing an informative standard evaluation task. In addition, we provide several baselines for multilingual sentence understanding, including two based on machine translation systems, and two that use parallel data to train aligned multilingual bag-of-words and LSTM encoders. We find that XNLI represents a practical and challenging evaluation suite, and that directly translating the test data yields the best performance among available baselines.

多语言机器翻译已被证明是一种有效的策略，可以用单个模型在多种语言之间进行翻译。但是，大多数研究都集中在多语言句子翻译上，而无需考虑跨不同语言生成长文档，这需要了解多语言上下文依赖性，并且通常更难。在本文中，我们首先是天真地纳入辅助多语言数据的辅助目标或源辅助数据对我们感兴趣的源目标对没有任何改进。在这一观察过程中，我们提出了一个名为多语言传递性（MTRAN）的新型框架，以在多语言模型中通过源辅助目标找到一个隐式的最佳途径。为了鼓励MTRANS，我们提出了一种称为三重平行数据（TPD）的新方法，该方法使用包含（源 - 载体，辅助目标和源目标）的平行三重线进行训练。然后，辅助语言充当枢轴，并自动促进隐式信息过渡流，从而更容易翻译。我们进一步提出了一个名为“双向多语言协议”（BI-Magree）的新颖框架，该框架鼓励不同语言之间的双向协议。为了鼓励Bi-Magree，我们提出了一种称为多语言Kullback-Leibler Divergence（MKL）的新颖方法，该方法迫使输入的输出分布具有相同的含义，但以不同的语言彼此一致。实验结果表明，我们的方法对三个文档翻译任务的强大基准进行了一致的改进：IWSLT2015 ZH-EN，DE-EN和VI-EN。我们的分析验证了MTRAN和BI-MAGREE的实用性和存在，我们的框架和方法对合成辅助数据有效。

多语言语言模型（\ mllms），如mbert，xlm，xlm-r，\ textit {etc。}已成为一种可行的选择，使预先估计到大量语言的力量。鉴于他们的成功在零射击转移学习中，在（i）建立更大的\ mllms〜覆盖了大量语言（ii）创建覆盖更广泛的任务和语言来评估的详尽工作基准mllms〜（iii）分析单音零点，零拍摄交叉和双语任务（iv）对Monolingual的性能，了解\ mllms〜（v）增强（通常）学习的通用语言模式（如果有的话）有限的容量\ mllms〜以提高他们在已见甚至看不见语言的表现。在这项调查中，我们审查了现有的文学，涵盖了上述与\ MLLMS有关的广泛研究领域。根据我们的调查，我们建议您有一些未来的研究方向。

Translating training data into many languages has emerged as a practical solution for improving cross-lingual transfer. For tasks that involve span-level annotations, such as information extraction or question answering, an additional label projection step is required to map annotated spans onto the translated texts. Recently, a few efforts have utilized a simple mark-then-translate method to jointly perform translation and projection by inserting special markers around the labeled spans in the original sentence. However, as far as we are aware, no empirical analysis has been conducted on how this approach compares to traditional annotation projection based on word alignment. In this paper, we present an extensive empirical study across 42 languages and three tasks (QA, NER, and Event Extraction) to evaluate the effectiveness and limitations of both methods, filling an important gap in the literature. Experimental results show that our optimized version of mark-then-translate, which we call EasyProject, is easily applied to many languages and works surprisingly well, outperforming the more complex word alignment-based methods. We analyze several key factors that affect end-task performance, and show EasyProject works well because it can accurately preserve label span boundaries after translation. We will publicly release all our code and data.

本文介绍了基于Wav2VEC 2.0的跨语言语音表示学习的大规模模型。我们在128种语言中培训最多2B个公共讲话音频的近半小时的型号的模型，比公共数据的数量级比最大的已知事先工作。我们的评估涵盖了广泛的任务，域，数据制度和语言，都是高低资源。在Covost-2语音翻译基准测试中，我们将先前的最先进的状态平均为7.4 BLEU超过21个翻译方向进入英语。对于语音识别，XLS-R在Babel，MLS，CommonVoice以及Voxpopuli上的最佳已知工作中提高，降低了相对的误差率14-34％。 XLS-R还在Voxlingua107语言识别上设置了新的技术状态。此外，我们表明，具有足够的模型规模，交叉思维预先预测可以在将英语演讲翻译成其他语言时才能优于英语撇印，这是一个有利于单晶的预借预制的设置。我们希望XLS-R可以帮助改善世界上更多语言的语音处理任务。

Open-Domain Generative Question Answering has achieved impressive performance in English by combining document-level retrieval with answer generation. These approaches, which we refer to as GenQA, can generate complete sentences, effectively answering both factoid and non-factoid questions. In this paper, we extend GenQA to the multilingual and cross-lingual settings. For this purpose, we first introduce GenTyDiQA, an extension of the TyDiQA dataset with well-formed and complete answers for Arabic, Bengali, English, Japanese, and Russian. Based on GenTyDiQA, we design a cross-lingual generative model that produces full-sentence answers by exploiting passages written in multiple languages, including languages different from the question. Our cross-lingual generative system outperforms answer sentence selection baselines for all 5 languages and monolingual generative pipelines for three out of five languages studied.

Much recent progress in applications of machine learning models to NLP has been driven by benchmarks that evaluate models across a wide variety of tasks. However, these broad-coverage benchmarks have been mostly limited to English, and despite an increasing interest in multilingual models, a benchmark that enables the comprehensive evaluation of such methods on a diverse range of languages and tasks is still missing. To this end, we introduce the Cross-lingual TRansfer Evaluation of Multilingual Encoders (XTREME) benchmark, a multi-task benchmark for evaluating the cross-lingual generalization capabilities of multilingual representations across 40 languages and 9 tasks. We demonstrate that while models tested on English reach human performance on many tasks, there is still a sizable gap in the performance of cross-lingually transferred models, particularly on syntactic and sentence retrieval tasks. There is also a wide spread of results across languages. We release the benchmark 1 to encourage research on cross-lingual learning methods that transfer linguistic knowledge across a diverse and representative set of languages and tasks.

在所有人类语言对之间实现通用翻译是机器翻译的圣杯（MT）研究。虽然最近在大量的多语言MT中的进展是达到这一目标的一步，但它变得明显，即简单地通过在更加平行数据上训练扩展多语言MT系统是不可编译的，因为用于低资源和非英语的标记数据的可用性 - 姓氏对禁止有限。为此，我们展示了一种务实的方法，可以使用监督和自我监督目标的混合来构建涵盖数百种语言的多语种MT模型，具体取决于不同语言对的数据可用性。我们展示这两种训练范例之间的协同作用使模型能够在零资源设置中产生高质量的翻译，甚至超过监控的用于中资和中资和中资质。我们开展广泛的实验，了解多语言监督，域错配和平行和单机数据量的效果，以了解我们自我监督的多语言模型的质量。为了展示方法的可扩展性，我们培训具有200多种语言的模型，并在几个先前研究的语言上展示了对零资源翻译的高性能。我们希望我们的调查结果将成为踏脚石，以便为下一千种语言进行翻译。