Wiki-40B: Multilingual Language Model Dataset

By Mandy Guo, Zihang Dai, Denny Vrandečić, Rami Al-Rfou. 2020

139 citations recorded by the source corpus.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Deduplicating Training Data Makes Language Models Better (2021, methodology): “Wikipedia (Wiki-40B) consists of multi-lingual cleaned Wikipedia text (Guo et al., 2020).”
Imitating Interactive Intelligence (2020, background): “For comparison, Figure 11C shows the agent produced word frequency versus those for a dataset constructed from Wikipedia (Guo et al., 2020).”
Counterfactual Memorization in Neural Language Models (2021, methodology): “We estimate and analyze counterfactual memorization of training examples in three standard text datasets: RealNews [Zellers et al., 2019], C4 [Raffel et al., 2020a] and Wiki40B:en [Guo et al., 2020].”
Transformer Quality in Linear Time (2022, methodology): “We train and evaluate all models with 2 18 tokens per batch for 125K steps, with context lengths ranging from 512 to 8K for Wiki-40B and 1K to 8K for PG-19.”
Combiner: Full Attention Transformer with Sparse Computation Cost (2021, methodology): “49 For language modeling, we focus on the Wiki-40BEn dataset [39], which consists of clean Wikipedia pages in English.”
Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities (2024, background): “For instance, web pages like Wikipedia [52] can contribute to a language model’s broad linguistic understanding.”
FRMT: A Benchmark for Few-Shot Region-Aware Machine Translation (2022, unknown): “3.0) by Guo et al. (2020”
Poisoning Web-Scale Training Datasets is Practical (2023, methodology): “this vulnerability, we investigate the Wiki-40B dataset [25] which is frequently used to train large multilingual models.”
Symbolic Discovery of Optimization Algorithms (2023, methodology): “Autoregressive language modeling We first experiment on two smaller-scale academic datasets Wiki40B (Guo et al., 2020) and PG-19 (Rae et al.”
MTG: A Benchmark Suite for Multilingual Text Generation (2021, background): “Wiki-40B (Guo et al., 2020) is a multilingual language model dataset across 40+ languages.”
A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation (2021, methodology): “Our raw data are sampled from English W IKI -40B (Guo et al., 2020) dataset.”