XenC: An Open-Source Tool for Data Selection in Natural Language Processing

By Anthony Rousseau. 2013

77 citations recorded by the source corpus.

Abstract In this paper we describe XenC, an open-source tool for data selection aimed at Natural Language Processing (NLP) in general and Statistical Machine Translation (SMT) or Automatic Speech Recognition (ASR) in particular. Usually, when building a SMT or ASR system, the considered task is related to a specific domain of application, like news articles or scientific talks for instance. The goal of XenC is to allow selection of relevant data regarding the considered task, which will be used to build the statistical models for such a system. It is done by computing the difference between cross-entropy scores of sentences from a large out-of-domain corpus and sentences from a corpus considered as in-domain for the task. Written in C++, this tool can operate on monolingual or bilingual data and is language-independent. XenC, now part of the LIUM toolchain for SMT, is actively developed since December 2011 and used in many MT projects.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

The NiuTrans Machine Translation Systems for WMT19 (2019, methodology): “We back-translated 10M monolingual in-domain data from the collection of NewsCrawl2016-2018 filtered by XenC (Rousseau, 2013).”
Zipporah: a Fast and Scalable Data Cleaning System for Noisy Web-Crawled Parallel Corpora (2017, methodology): “XenC (Rousseau, 2013), an open-source tool, also selects data based on cross-entropy scores on language models.”
Learn Spelling from Teachers: Transferring Knowledge from Language Models to Sequence-to-Sequence Speech Recognition (2019, methodology): “We used transformer based Seq2Seq as backbone, and conducted experiments on public available Chinese datasets AISHELL-1 (speech) and CLMAD (external text).”
LIMSI@WMT’16: Machine Translation of News (2016, methodology): “Instead of directly training an LM on these corpora, we extracted from them in-domain sentences using the Moore-Lewis (Moore and Lewis, 2010) ﬁltering method, more speciﬁcally its implementation in XenC (Rousseau, 2013).”
CUNI Submissions in WMT18 (2018, methodology): “We also used the genuine parallel data extracted from CzEng 1.6 (Bojar et al., 2016a) using the XenC toolkit (Rousseau, 2013) with Czech monolingual news corpus as the reference in-domain text.”
The JHU Parallel Corpus Filtering Systems for WMT 2018 (2018, unknown): “XenC (Rousseau, 2013), an open-source tool, also selects data based on crossentropy scores on language models.”
Collecting Resources in Sub-Saharan African Languages for Automatic Speech Recognition: a Case Study of Wolof (2016, methodology): “Xenc: An open-source tool for data selection in natural language processing.”
Sheffield Systems for the English-Romanian WMT Translation Task (2016, background): “1 (Rousseau, 2013) in mode 2 3 with the parallel corpora (Europarl7, SETimes2) as in-domain data.”
Accelerating Text Communication via Abbreviated Sentence Input (2021, methodology): “We created a test and development set using data collected on touchscreen phones (Vertanen et al., 2015, 2013) and watches (Vertanen et al., 2018, 2019).”
The repetition rate of text as a predictor of the effectiveness of machine translation adaptation (2014, methodology): “…6 (Bojar et al., 2013)) by using the in-domain text as seed in the method proposed by Axelrod et al. (2011) and available within the XenC toolkit (Rousseau, 2013); this was done to augment the amount of training data, since the size of in-domain text available for that language pair (15.4/17.9…”
Enhancing the TED-LIUM Corpus with Selected Data for Language Modeling and More TED Talks (2014, methodology): “In order to select relevant data to the task of performing Automatic Speech Recognition on TED Talks, we used a data selection approach described in (Moore and Lewis, 2010), which we implemented in an open-source tool named XenC we developed and released last year (Rousseau, 2013).”
The 2016 BBN Georgian telephone speech keyword spotting system (2017, methodology): “We post-process the downloaded web text using the XenC tool[20], which selects high-confidence text based on the cross-entropy between training data and the provided web text.”
LIUM Machine Translation Systems for WMT17 News Translation Task (2017, methodology): “tences from the czeng EU corpus applying modified Moore-Lewis filtering with XenC (Rousseau, 2013).”
Combined spoken language translation (2014, methodology): “Data selection was performed by means of XenC toolkit [57] exploiting bilingual cross-entropy difference [58] separately for each available training corpus except the in-domain WIT3 data.”
The QT21/HimL Combined Machine Translation System (2016, unknown): “We select in-domain sentences from the latter using the Moore-Lewis (Moore and Lewis, 2010) ﬁltering method, more speciﬁcally its implementation in XenC (Rousseau, 2013).”