Linguistically-augmented perplexity-based data selection for language models

By Antonio Toral, Pavel Pecina, Longyue Wang, Josef van Genabith. 2015

19 citations recorded by the source corpus.

HighlightsWord-level linguistic information for perplexity-based data selection.Evaluation and analysis for four languages: English, Spanish, Czech and Chinese.Combination of models lead to lower perplexity than the state-of-the-art baseline. This paper explores the use of linguistic information for the selection of data to train language models. We depart from the state-of-the-art method in perplexity-based data selection and extend it in order to use word-level linguistic units (i.e. lemmas, named entity categories and part-of-speech tags) instead of surface forms. We then present two methods that combine the different types of linguistic knowledge as well as the surface forms (1, naive selection of the top ranked sentences selected by each method; 2, linear interpolation of the datasets selected by the different methods). The paper presents detailed results and analysis for four languages with different levels of morphologic complexity (English, Spanish, Czech and Chinese). The interpolation-based combination outperforms the purely statistical baseline in all the scenarios, resulting in language models with lower perplexity. In relative terms the improvements are similar regardless of the language, with perplexity reductions achieved in the range 7.72-13.02%. In absolute terms the reduction is higher for languages with high type-token ratio (Chinese, 202.16) or rich morphology (Czech, 81.53) and lower for the remaining languages, Spanish (55.2) and English (34.43 on the English side of the same parallel dataset as for Czech and 61.90 on the same parallel dataset as for Spanish).

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Douwe Kiela
2. Mohit Bansal
3. Nazneen Rajani
4. Tristan Thrush
5. Yacine Jernite
6. Yejin Choi
7. Margaret Mitchell
8. Angelina McMillan-Major

In Large Language Models

1. Tristan Thrush
2. Wolfgang Menzel
3. Angelina McMillan-Major
4. Douwe Kiela
5. Yacine Jernite
6. Alexandra Sasha Luccioni
7. Mirela-Stefania Duma
8. Margaret Mitchell

In Natural Language Processing

1. Tristan Thrush
2. Douwe Kiela
3. Nazneen Rajani
4. Wolfgang Menzel
5. Angelina McMillan-Major
6. Yacine Jernite
7. Alexandra Sasha Luccioni
8. Mirela-Stefania Duma

Citations in the corpus

• Measuring Data (2022, methodology): “It was subsequently leveraged to evaluate the performance of language models (Brown et al., 1992; Bengio et al., 2000; Melis et al., 2017) as well as datasets, for tasks such as perplexity sampling (De la Rosa et al., 2022) and data selection (Toral et al., 2015).”
• Data Selection for IT Texts using Paragraph Vector (2016, background): “Toral et al. (2015) introduced linguistic information such as lemmas, named entities and part-of-speech tags into the preprocessing of the data and then ranked the sentences by per-plexity.”