我们提出一个免费的日语平行语料库。它包括1500万个对齐段，并通过编译和过滤几种现有资源来获得。在本文中，我们描述了现有资源，其数量和质量，我们应用的过滤以提高语料库的质量以及现成的语料库的内容。我们还通过训练和评估一些标准的MT系统来评估该语料库的实用性和过滤质量。

我们介绍了第一个用于濒危Erzya语言与俄语以及我们为训练和评估它收集的数据集的神经机器翻译系统。BLEU分别分别为Erzya和Russian的BLEU分数分别为17和19，其中一半以上的翻译被以母语为母语的人可以接受。我们还调整了模型以在Erzya和其他10种语言之间转换，但是如果没有其他并行数据，这些方向上的质量仍然很低。我们将翻译模型与收集的文本语料库一起发布，新的语言标识模型以及适合Erzya语言的多语言句子编码器。这些资源将在https://github.com/slone-nlp/myv-nmt上找到。

语言模型预训练的最新进展利用大规模数据集创建多语言模型。但是，这些数据集中大多遗漏了低资源语言。这主要是因为网络上没有很好地表示口语，因此被排除在用于创建数据集的大规模爬网中。此外，这些模型的下游用户仅限于最初选择用于预训练的语言的选择。这项工作调查了如何最佳利用现有的预培训模型来为16种非洲语言创建低资源翻译系统。我们关注两个问题：1）如何将预训练的模型用于初始预培训中未包含的语言？ 2）生成的翻译模型如何有效地转移到新域？为了回答这些问题，我们创建了一个新的非洲新闻语料库，涵盖16种语言，其中8种语言不属于任何现有评估数据集的一部分。我们证明，将两种语言转移到其他语言和其他领域的最有效策略是，以少量的高质量翻译数据微调大型预训练模型。

The Annals of Joseon Dynasty (AJD) contain the daily records of the Kings of Joseon, the 500-year kingdom preceding the modern nation of Korea. The Annals were originally written in an archaic Korean writing system, `Hanja', and were translated into Korean from 1968 to 1993. The resulting translation was however too literal and contained many archaic Korean words; thus, a new expert translation effort began in 2012. Since then, the records of only one king have been completed in a decade. In parallel, expert translators are working on English translation, also at a slow pace and produced only one king's records in English so far. Thus, we propose H2KE, a neural machine translation model, that translates historical documents in Hanja to more easily understandable Korean and to English. Built on top of multilingual neural machine translation, H2KE learns to translate a historical document written in Hanja, from both a full dataset of outdated Korean translation and a small dataset of more recently translated contemporary Korean and English. We compare our method against two baselines: a recent model that simultaneously learns to restore and translate Hanja historical document and a Transformer based model trained only on newly translated corpora. The experiments reveal that our method significantly outperforms the baselines in terms of BLEU scores for both contemporary Korean and English translations. We further conduct extensive human evaluation which shows that our translation is preferred over the original expert translations by both experts and non-expert Korean speakers.

We present Charles University submissions to the WMT22 General Translation Shared Task on Czech-Ukrainian and Ukrainian-Czech machine translation. We present two constrained submissions based on block back-translation and tagged back-translation and experiment with rule-based romanization of Ukrainian. Our results show that the romanization only has a minor effect on the translation quality. Further, we describe Charles Translator, a system that was developed in March 2022 as a response to the migration from Ukraine to the Czech Republic. Compared to our constrained systems, it did not use the romanization and used some proprietary data sources.

The BERT family of neural language models have become highly popular due to their ability to provide sequences of text with rich context-sensitive token encodings which are able to generalise well to many NLP tasks. We introduce gaBERT, a monolingual BERT model for the Irish language. We compare our gaBERT model to multilingual BERT and the monolingual Irish WikiBERT, and we show that gaBERT provides better representations for a downstream parsing task. We also show how different filtering criteria, vocabulary size and the choice of subword tokenisation model affect downstream performance. We compare the results of fine-tuning a gaBERT model with an mBERT model for the task of identifying verbal multiword expressions, and show that the fine-tuned gaBERT model also performs better at this task. We release gaBERT and related code to the community.

本文介绍了对土耳其语可用于的语料库和词汇资源的全面调查。我们审查了广泛的资源，重点关注公开可用的资源。除了提供有关可用语言资源的信息外，我们还提供了一组建议，并确定可用于在土耳其语言学和自然语言处理中进行研究和建筑应用的数据中的差距。

机器翻译系统（MTS）是通过将文本或语音从一种语言转换为另一种语言的有效工具。在像印度这样的大型多语言环境中，对有效的翻译系统的需求变得显而易见，英语和一套印度语言（ILS）正式使用。与英语相反，由于语料库的不可用，IL仍然被视为低资源语言。为了解决不对称性质，多语言神经机器翻译（MNMT）系统会发展为在这个方向上的理想方法。在本文中，我们提出了一个MNMT系统，以解决与低资源语言翻译有关的问题。我们的模型包括两个MNMT系统，即用于英语印度（一对多），另一个用于指示英语（多一对多），其中包含15个语言对（30个翻译说明）的共享编码器码头。由于大多数IL对具有很少的平行语料库，因此不足以训练任何机器翻译模型。我们探索各种增强策略，以通过建议的模型提高整体翻译质量。最先进的变压器体系结构用于实现所提出的模型。大量数据的试验揭示了其优越性比常规模型的优势。此外，本文解决了语言关系的使用（在方言，脚本等方面），尤其是关于同一家族的高资源语言在提高低资源语言表现方面的作用。此外，实验结果还表明了ILS的倒退和域适应性的优势，以提高源和目标语言的翻译质量。使用所有这些关键方法，我们提出的模型在评估指标方面比基线模型更有效，即一组ILS的BLEU（双语评估研究）得分。

In this paper we present two datasets for Tamasheq, a developing language mainly spoken in Mali and Niger. These two datasets were made available for the IWSLT 2022 low-resource speech translation track, and they consist of collections of radio recordings from the Studio Kalangou (Niger) and Studio Tamani (Mali) daily broadcast news. We share (i) a massive amount of unlabeled audio data (671 hours) in five languages: French from Niger, Fulfulde, Hausa, Tamasheq and Zarma, and (ii) a smaller parallel corpus of audio recordings (17 hours) in Tamasheq, with utterance-level translations in the French language. All this data is shared under the Creative Commons BY-NC-ND 3.0 license. We hope these resources will inspire the speech community to develop and benchmark models using the Tamasheq language.

我们介绍Samanantar，是最大的公开可用的并行Corpora Collection，用于指示语言。该集合中的英语和11个上线语言之间总共包含4970万句对（来自两种语言系列）。具体而言，我们从现有的公共可用并行基层编译1240万句对，另外，从网络上挖掘3740万句对，导致4倍增加。我们通过组合许多语料库，工具和方法来挖掘网站的并行句子：（a）Web爬行单格式语料库，（b）文档OCR，用于从扫描的文档中提取句子，（c）用于对齐句子的多语言表示模型，以及（d）近似最近的邻居搜索搜索大量句子。人类评估新矿业的Corpora的样本验证了11种语言的高质量平行句子。此外，我们使用英语作为枢轴语言，从英式并行语料库中提取所有55个指示语言对之间的834百万句子对。我们培训了跨越Samanantar上所有这些语言的多语种NMT模型，这在公开可用的基准上表现出现有的模型和基准，例如弗洛雷斯，建立萨曼塔尔的效用。我们的数据和模型可在Https://indicnlp.ai4bharat.org/samanantar/上公开提供，我们希望他们能够帮助推进NMT和Multibingual NLP的研究。

文本归一化是缺乏严格拼写惯例的低资源语言的至关重要技术。迄今为止，低资源的文本归一化依赖于手工制作的规则，这些规则被认为比神经方法更有效。在本文中，我们研究了Ligurian（一种濒临灭绝的浪漫语言）的文本正常化情况。我们收集了4,394个Ligurian句子，并配对其标准化版本，也是Ligurian的第一个单语语料库。我们表明，尽管有少量可用的数据，但可以训练基于紧凑的变压器的模型，以通过使用反射和适当的令牌化来达到非常低的错误率。我们的数据集向公众发布。

本文介绍了Persian的HMBLOGS语料库，作为一种低资源语言。此语料库已根据波斯博客的空间的一段时间内基于大约15岁的近2000万博客帖子编制的，包括超过68亿令牌。可以声称，此语料库目前是最大的波斯语料库，这些语料库是为波斯语而独立制定的。该语料库以原始和预处理的形式呈现，并且基于预处理的语料库，产生了一些单词嵌入模型。通过提供的模型，将HMBlogs与波斯中可用的一些最重要的公司进行比较，结果表明了HMBLOGS语料库的优势。这些评估还提供了语料库，评估数据集，模型生产方法，不同的高参数甚至评估方法的重要性和影响。除了评估语料库及其产生的语言模型之外，该研究还提供了一个语义类比数据集。

Machine Translation (MT) system generally aims at automatic representation of source language into target language retaining the originality of context using various Natural Language Processing (NLP) techniques. Among various NLP methods, Statistical Machine Translation(SMT). SMT uses probabilistic and statistical techniques to analyze information and conversion. This paper canvasses about the development of bilingual SMT models for translating English to fifteen low-resource Indian Languages (ILs) and vice versa. At the outset, all 15 languages are briefed with a short description related to our experimental need. Further, a detailed analysis of Samanantar and OPUS dataset for model building, along with standard benchmark dataset (Flores-200) for fine-tuning and testing, is done as a part of our experiment. Different preprocessing approaches are proposed in this paper to handle the noise of the dataset. To create the system, MOSES open-source SMT toolkit is explored. Distance reordering is utilized with the aim to understand the rules of grammar and context-dependent adjustments through a phrase reordering categorization framework. In our experiment, the quality of the translation is evaluated using standard metrics such as BLEU, METEOR, and RIBES

我们提出了多语言开放文本（MOT），这是一种新的多语言语料库，其中包含44种语言的文本，其中许多语言限制了现有的文本资源用于自然语言处理。该语料库的第一个版本包含超过280万篇新闻文章，并在2001 - 2022年之间发表了另外100万个短片段（照片标题，视频描述等），并从美国之声网站收集。我们描述了收集，过滤和处理数据的过程。原始材料在公共领域，我们的收藏品使用Creative Commons许可证（CC By 4.0）获得许可，并且用于创建该语料库的所有软件均在MIT许可证下发布。随着其他文档的发布，该语料库将定期更新。

本文记录了伊图哥本哈根（ITU Copenhagen）生产的法罗伊斯（Faroese）和丹麦（Faroese）之间的句子对数据集。数据涵盖了两种源语言的tranlsation，旨在用作此语言对的机器翻译系统的培训数据。

This paper presents the OPUS ecosystem with a focus on the development of open machine translation models and tools, and their integration into end-user applications, development platforms and professional workflows. We discuss our on-going mission of increasing language coverage and translation quality, and also describe on-going work on the development of modular translation models and speed-optimized compact solutions for real-time translation on regular desktops and small devices.

In this work, we introduce IndicXTREME, a benchmark consisting of nine diverse tasks covering 18 languages from the Indic sub-continent belonging to four different families. Across languages and tasks, IndicXTREME contains a total of 103 evaluation sets, of which 51 are new contributions to the literature. To maintain high quality, we only use human annotators to curate or translate\footnote{for IndicXParaphrase, where an automatic translation system is used, a second human verification and correction step is done.} our datasets. To the best of our knowledge, this is the first effort toward creating a standard benchmark for Indic languages that aims to test the zero-shot capabilities of pretrained language models. We also release IndicCorp v2, an updated and much larger version of IndicCorp that contains 20.9 billion tokens in 24 languages. We pretrain IndicBERT v2 on IndicCorp v2 and evaluate it on IndicXTREME to show that it outperforms existing multilingual language models such as XLM-R and MuRIL.

神经机器翻译（NMT）模型在大型双语数据集上已有效。但是，现有的方法和技术表明，该模型的性能高度取决于培训数据中的示例数量。对于许多语言而言，拥有如此数量的语料库是一个牵强的梦想。我们从单语言词典探索新语言的单语扬声器中汲取灵感，我们研究了双语词典对具有极低或双语语料库的语言的适用性。在本文中，我们使用具有NMT模型的双语词典探索方法，以改善资源极低的资源语言的翻译。我们将此工作扩展到多语言系统，表现出零拍的属性。我们详细介绍了字典质量，培训数据集大小，语言家族等对翻译质量的影响。多种低资源测试语言的结果表明，我们的双语词典方法比基线相比。

Language identification (LID) is a crucial precursor for NLP, especially for mining web data. Problematically, most of the world's 7000+ languages today are not covered by LID technologies. We address this pressing issue for Africa by introducing AfroLID, a neural LID toolkit for $517$ African languages and varieties. AfroLID exploits a multi-domain web dataset manually curated from across 14 language families utilizing five orthographic systems. When evaluated on our blind Test set, AfroLID achieves 95.89 F_1-score. We also compare AfroLID to five existing LID tools that each cover a small number of African languages, finding it to outperform them on most languages. We further show the utility of AfroLID in the wild by testing it on the acutely under-served Twitter domain. Finally, we offer a number of controlled case studies and perform a linguistically-motivated error analysis that allow us to both showcase AfroLID's powerful capabilities and limitations.

近年来，基于变压器的模型已导致自然语言处理的语言建模取得重大进步。但是，他们需要大量的数据接受（预先）训练，并且除英语以外的语言中缺乏语料库。最近，一些计划提出了从自动网络爬行获得的多语言数据集。但是，西班牙语的结果具有重要的缺点，因为与其他语言相比，它们要么太小，要么呈现出较低的质量，从而获得了次优的清洁和重复数据删除。在本文中，我们介绍了Escorpius，这是一种西班牙爬行语料库，该语料库是从附近的1 pb普通爬网数据中获得的。它是西班牙语中最广泛的语料库，其提取，纯化和重复数据删除的质量水平。我们的数据策划过程涉及一条新型的高度平行清洁管道，并包含一系列重复数据删除机制，以确保文档和段落边界的完整性。此外，我们同时维护源网页URL和WARC Shard Origin URL，以抱怨欧盟法规。 Escorpius已根据CC BY-NC-ND 4.0许可发布，可在HuggingFace上获得。