Evaluating the Robustness of Neural Language Models to Input Perturbations

By Milad Moradi, Matthias Samwald. 2021

142 citations recorded by the source corpus.

High-performance neural language models have obtained state-of-the-art results on a wide range of Natural Language Processing (NLP) tasks. However, results for common benchmark datasets often do not reflect model reliability and robustness when applied to noisy, real-world data. In this study, we design and implement various types of character-level and word-level perturbation methods to simulate realistic scenarios in which input texts may be slightly noisy or different from the data distribution on which NLP systems were trained. Conducting comprehensive experiments on different NLP tasks, we investigate the ability of high-performance language models such as BERT, XLNet, RoBERTa, and ELMo in handling different types of input perturbations. The results suggest that language models are sensitive to input perturbations and their performance can decrease even when small changes are introduced. We highlight that models need to be further improved and that current benchmarks are not reflecting model robustness well. We argue that evaluations on perturbed inputs should routinely complement widely-used benchmarks in order to yield a more realistic understanding of NLP systems’ robustness.

Subfields: Artificial Intelligence, Information Systems, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Safety, Robustness & Interpretability

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “To that end, Moradi and Samwald (2021) show that models are very brittle”
On Evaluating Adversarial Robustness of Large Vision-Language Models (2023, background): “…flexible (e.g., beyond word replacement) and semantically preserving methods are proposed to produce natural language adversarial examples [9, 49, 50, 52, 53, 70, 78, 102, 104, 110], as well as benchmarks and datasets to more thoroughly evaluate the adversarial robustness of LMs [56, 90–92].”
The Fallacy of AI Functionality (2022, methodology): “For example, when large language models are used to process the queries of AI-powered web search [131], the models’ fragility to misspellings [125, 147], or trivial changes to format [19] and context [18] can lead to unexpected results.”
Evaluation Gaps in Machine Learning Practice (2022, methodology): “Our framework for analyzing the gaps builds upon and complements other streams of work on ML evaluation practices, including addressing distribution shifts between development data and application data [34, 94, 160], and robustness to perturbations in test items [118, 132, 173 ?”
GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts (2023, background): “Furthermore, their widespread use has made them targets for adversarial attacks, including backdoor attacks [5, 32, 41], prompt injection [23,36,49], and data poisoning [34,43,73].”
Prompt Injection attack against LLM-integrated Applications (2023, background): “However, this expanded usage is accompanied by heightened security vulnerabilities, manifested by a broad spectrum of adversarial tactics such as jailbreak [15, 41, 60] and backdoor [7, 36, 68], and complex data poisoning [32, 38, 67].”
Out of Style: RAG's Fragility to Linguistic Variation (2025, unknown): “Prior research has examined impacts of syntactic perturbations (Moradi and Samwald, 2021; Singh et al., 2024), round-trip translation (Bhandari and Chen, 2023), politeness variations (Yin et al., 2024), equivalent queries (Cao et al., 2024) and scale of model size on robustness to grammatical…”