Intelligent Selection of Language Model Training Data

By Robert C. Moore, William D. Lewis. 2010

671 citations recorded by the source corpus.

Subfields: Large Language Models, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

In Optimization & Training

Citations in the corpus

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation (2014, methodology): “…SMT framework introduced in (Koehn et al., 2003) and (Marcu and Wong, 2002), the translation model log p(e | f) is factorized into the translation probabilities of matching phrases in the source and target sentences.2 These probabilities are once again considered additional features in the…”
Edinburgh’s Phrase-based Machine Translation Systems for WMT-14 (2014, methodology): “We also experimented with modified MooreLewis (Moore and Lewis, 2010; Axelrod et al., 2011) data selection, using the EMEA corpus as the in-domain corpus (for the language model required in MML) and selecting from all the out-ofdomain data.”
Achieving Human Parity on Automatic Chinese to English News Translation (2018, methodology): “The de-facto standard method for data selection for SMT is [27] and [2].”
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets (2021, background): “, 2020) and downstream task performance (Moore and Lewis, 2010; Xu and Koehn, 2017; Khayrallah and Koehn, 2018; Brown et al., 2020).”
Fast Domain Adaptation for Neural Machine Translation (2016, methodology): “In SMT, this has been done for language model training [Gao et al., 2002, Moore and Lewis, 2010] and translation model training [Matsoukas et al., 2009, Foster et al., 2010, Axelrod et al., 2011, Mansour and Ney, 2014].”
Zipporah: a Fast and Scalable Data Cleaning System for Noisy Web-Crawled Parallel Corpora (2017, background): “Moore and Lewis (2010) computed cross-entropy between in-domain and out-of-domain language models to select data for training language models.”
Learning to select data for transfer learning with Bayesian Optimization (2017, methodology): “While this has been shown to work empirically (Moore and Lewis, 2010; Axelrod et al., 2011; Plank and van Noord, 2011; Van Asch and Daele-mans, 2010; Remus, 2012), using a pre-existing metric leaves us unable to adapt to the characteristics of our task T and target domain D T and fore-goes…”
Federated Learning of N-Gram Language Models (2019, background): “The highest quality n-gram models are trained over data that are well-matched to the desired output (Moore and Lewis, 2010).”
Dynamic Data Selection for Neural Machine Translation (2017, methodology): “In PBMT, this issue has been addressed by applying intelligent data selection, and it has consistently been shown that using more data does not always improve translation quality (Moore and Lewis, 2010; Axelrod et al., 2011; Gasc´o et al., 2012).”
Extreme Adaptation for Personalized Neural Machine Translation (2018, methodology): “Domain adaptation techniques for MT often rely on data selection (Moore and Lewis, 2010; Li et al., 2010; Chen et al., 2017; Wang et al., 2017), tuning (Luong and Manning, 2015; Miceli Barone et al., 2017), or adding domain tags to NMT input (Chu et al., 2017).”
DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining (2023, background): “Moore-Lewis selection [32; 3; 15] selects examples with high cross-entropy difference (similar to excess log-perplexity) between language models trained on target and raw data.”
State-of-the-Art Deep Learning: Evolving Machine Intelligence Toward Tomorrow’s Intelligent Network Traffic Control Systems (2017, background): “For instance, consider the intelligence translation area, which confronts the challenge in determining how to select non-domain-speciﬁc language model training data [215].”
The RWTH Aachen German-English Machine Translation System for WMT 2014 (2014, methodology): “The monolingual data selection is based on crossentropy difference as described in (Moore and Lewis, 2010).”
Alibaba Submission to the WMT18 Parallel Corpus Filtering Task (2018, methodology): “The language model is successfully used to select domain-related corpus (Yasuda et al., 2008; Moore and Lewis, 2010).”