Manipulating SGD with Data Ordering Attacks

By Ilia Shumailov, Zakhar Shumaylov, Dmitry Kazhdan, Yiren Zhao, Nicolas Papernot, Murat A. Erdogdu, Ross Anderson. 2021

122 citations recorded by the source corpus.

Machine learning is vulnerable to a wide variety of attacks. It is now well understood that by changing the underlying data distribution, an adversary can poison the model trained with it or introduce backdoors. In this paper we present a novel class of training-time attacks that require no changes to the underlying dataset or model architecture, but instead only change the order in which data are supplied to the model. In particular, we find that the attacker can either prevent the model from learning, or poison it to learn behaviours specified by the attacker. Furthermore, we find that even a single adversarially-ordered epoch can be enough to slow down model learning, or even to reset all of the learning progress. Indeed, the attacks presented here are not specific to the model or dataset, but rather target the stochastic nature of modern learning procedures. We extensively evaluate our attacks on computer vision and natural language benchmarks to find that the adversary can disrupt model training and even introduce backdoors.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability, Privacy & Security

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Martin Jaggi
2. Sebastian U. Stich
3. Peter Richtárik
4. Zhifeng Li
5. Shuaiwen Leon Song
6. Donglin Zhuang
7. Xingyao Zhang
8. Yong Jiang

In Safety, Robustness & Interpretability

1. Yong Jiang
2. Baoyuan Wu
3. Zhifeng Li
4. Martin Jaggi
5. Shutao Xia
6. Yiming Li
7. Sara Hooker

In Privacy & Security

1. Martin Jaggi

Citations in the corpus

• Characterizing & Finding Good Data Orderings for Fast Convergence of Sequential Gradient Methods (2022, background): “[28] show that an adversary can slow down the training by choosing a bad order of data.”
• Randomness In Neural Network Training: Characterizing The Impact of Tooling (2021, background): “Even when batching is not used (all data is processed in a single batch), a difference in ordering can introduce stochasticity that may introduce security vulnerabilities ((Shumailov et al., 2021)).”
• Backdoor Learning: A Survey (2020, background): “Most recently, Shumailov et al. [50] proposed to inject hidden backdoors via manipulating the order of training samples without changing samples.”