The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation

By Orevaoghene Ahia, Julia Kreutzer, Sara Hooker. 2021

64 citations recorded by the source corpus.

A"bigger is better"explosion in the number of parameters in deep neural networks has made it increasingly challenging to make state-of-the-art networks accessible in compute-restricted environments. Compression techniques have taken on renewed importance as a way to bridge the gap. However, evaluation of the trade-offs incurred by popular compression techniques has been centered on high-resource datasets. In this work, we instead consider the impact of compression in a data-limited regime. We introduce the term low-resource double bind to refer to the co-occurrence of data limitations and compute resource constraints. This is a common setting for NLP for low-resource languages, yet the trade-offs in performance are poorly studied. Our work offers surprising insights into the relationship between capacity and generalization in data-limited regimes for the task of machine translation. Our experiments on magnitude pruning for translations from English into Yoruba, Hausa, Igbo and German show that in low-resource regimes, sparsity preserves performance on frequent sentences but has a disparate impact on infrequent ones. However, it improves robustness to out-of-distribution shifts, especially for datasets that are very distinct from the training distribution. Our findings suggest that sparsity can play a beneficial role at curbing memorization of low frequency attributes, and therefore offers a promising solution to the low-resource double bind.

Subfields: Artificial Intelligence, Natural Language Processing, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Efficiency & Compression

Citations in the corpus

Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “These can include choices about many stages of the optimization process [237, 129]; privacy constraints [24], widely used compression techniques [109, 15, 169] and the choice hardware [273] have all been found to amplify harm on underrepresented protected attributes [28].”
Intriguing Properties of Compression on Multilingual Models (2022, background): “pressed and dense models further find that compression may lead to erosion of performance on “challenging” samples and poor generalization (Ahia et al., 2021; Du et al., 2021a; Xu et al., 2021), a finding that we expand on and connect to language resourcedness.”
Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model (2024, background): “…future work explores compression techniques (Treviso et al., 2023) such as quantization (Dettmers et al., 2022; Frantar et al., 2022; Ahmadian et al., 2023) and pruning (Frantar and Alistarh, 2023; Ogueji et al., 2022; Gale et al., 2019; Ahia et al., 2021) for multilingual instruction-tuned models.”
Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold (2022, background): “Another example of this is Ahia et al. (2021), who show that for low-resource languages, weight pruning hurts performance on tail phenomena, but improves robustness to out-of-distribution shifts—this is not observed in the SQUARE ONE (high-resource) regime.”
Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamics (2022, background): “…et al., 2019a;b), atypical (Hooker et al., 2020; Buolamwini and Gebru, 2018; Hashimoto et al., 2018; S(cid:32)lowik and Bottou, 2021), challenging (Ahia et al., 2021; Baldock et al., 2021; Paul et al., 2021; Agarwal et al., 2021), prototypical or core subset selection (Paul et al., 2021; Sener…”