A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages

By Pedro Ortiz Suarez, Laurent Romary, Benoît Sagot. 2020

281 citations recorded by the source corpus.

We use the multilingual OSCAR corpus, extracted from Common Crawl via language classification, filtering and cleaning, to train monolingual contextualized word embeddings (ELMo) for five mid-resource languages. We then compare the performance of OSCAR-based and Wikipedia-based ELMo embeddings for these languages on the part-of-speech tagging and parsing tasks. We show that, despite the noise in the Common-Crawl-based OSCAR data, embeddings trained on OSCAR perform much better than monolingual embeddings trained on Wikipedia. They actually equal or improve the current state of the art in tagging and parsing for all five languages. In particular, they also improve over multilingual Wikipedia-based contextual embeddings (multilingual BERT), which almost always constitutes the previous state of the art, thereby showing that the benefit of a larger, more diverse corpus surpasses the cross-lingual benefit of multilingual embedding architectures.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (2021, background): “…recruited 51 volunteers speaking 70 languages to judge whether ﬁve publicly available multilingual web-crawled corpora (El-Kishky et al., 2020; Xue et al., 2021; Ortiz Suárez et al., 2020; Bañón et al., 2020; Schwenk et al., 2019) contain text in languages they report, as well as their quality.”
The ROOTS Search Tool: Data Transparency for LLMs (2023, background): “To date, there were several efforts to collect and clean large English and multilingual corpora (Raffel et al., 2020; Xue et al., 2021; Gao et al., 2020; Ortiz Suárez et al., 2020; Bañón et al., 2020; El-Kishky et al., 2020).”
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023, methodology): “…a community-selected and documented list of language data sources and its collection process is described in section 2, and 38% consists of text extracted from a pre-processed web crawl, OSCAR (Ortiz Suárez et al. (2020)), filtered with the help of native speakers, which is described in section 3.”
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets (2021, methodology): “OSCAR (Ortiz Suárez et al., 2019, 2020) is a set of monolingual corpora extracted from Common Crawl snapshots, specifically from the plain text WET format distributed by Common Crawl which removes all the HTML tags and converts the text formatting to UTF-8.”
Building Machine Translation Systems for the Next Thousand Languages (2022, background): “…(Esplà et al., 2019; Bañón et al., 2020), WikiMatrix (Schwenk et al., 2019), CCNET (Wenzek et al., 2020) and CCAligned (El-Kishky et al., 2020), OSCAR (Ortiz Suárez et al., 2019; Ortiz Suárez et al., 2020; Abadji et al., 2022), and several others; all of which have between 100 and 300 languages.”
How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models (2020, methodology): “IndoBERT (Wilie et al., 2020) was trained on a ∼3.6B word corpus collected from 15 different sources, which also include the Indonesian OSCAR corpus (Ortiz Suárez et al., 2020), an Indonesian Wiki dump, and data from Twitter.”
Documenting the English Colossal Clean Crawled Corpus (2021, background): “…recruited 51 volunteers speaking 70 languages to judge whether ﬁve publicly available multilingual web-crawled corpora (El-Kishky et al., 2020; Xue et al., 2020; Ortiz Su´arez et al., 2020; Ba˜n´on et al., 2020; Schwenk et al., 2019) contain text in languages they report, as well as their quality.”
Machine Translation into Low-resource Language Varieties (2021, methodology): “To simulate low-resource scenarios, we sample 10K, 100K and 1M UK sentences from the CoNLL’17 shared task and BE sentences from the OSCAR corpus (Or-tiz Suárez et al., 2020).”
MURI: High-Quality Instruction Tuning Datasets for Low-Resource Languages via Reverse Instructions (2024, unknown): “The core idea is summarized in Figure 1.”
OBELISC: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents (2023, methodology): “Our results demonstrate that the texts in OBELISC have a significantly lower average perplexity compared to the texts in c4 (Raffel et al., 2019), mmc4 (Zhu et al., 2023), and OSCAR (Ortiz Suárez et al., 2020).”
Ungoliant: An Optimized Pipeline for the Generation of a Very Large-Scale Multilingual Web Corpus (2021, unknown): “In this paper we propose a new multilingual corpus for language modeling, and for that we take inspiration in the OSCAR corpus and its pipeline goclassy 2 (Ortiz Su´arez et al., 2019, 2020), but we propose a new pipeline Ungoliant 3 that is faster, modular, parametrizable and well-documented.”
CoVoST 2: A Massively Multilingual Speech-to-Text Translation Corpus (2020, methodology): “For language model perplexity checks, 20M lines are sampled from the OSCAR corpus (Ortiz Suárez et al., 2020) for each CoVoST 2 language, except for English, Russian for which pre-trained language models (Ng et al., 2019) are utilized1.”
The Large Language Model GreekLegalRoBERTa (2024, unknown): “These four diﬀerent LLMs were trained on the dataset available in the Nomothesia platform [10], the Greek Parliament Proceedings [13], the Greek version of European Parliament Pro-ceedings Parallel Corpus [19], the Greek part of European Union legislation [6], Raptarchis [27] and OSCAR [26].”