Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models

By Beidi Chen, Tri Dao, Kaizhao Liang, Jiaming Yang, Zhao Song, Atri Rudra, Christopher R'e. 2021

95 citations recorded by the source corpus.

Overparameterized neural networks generalize well but are expensive to train. Ideally, one would like to reduce their computational cost while retaining their generalization benefits. Sparse model training is a simple and promising approach to achieve this, but there remain challenges as existing methods struggle with accuracy loss, slow training runtime, or difficulty in sparsifying all model components. The core problem is that searching for a sparsity mask over a discrete set of sparse matrices is difficult and expensive. To address this, our main insight is to optimize over a continuous superset of sparse matrices with a fixed structure known as products of butterfly matrices. As butterfly matrices are not hardware efficient, we propose simple variants of butterfly (block and flat) to take advantage of modern hardware. Our method (Pixelated Butterfly) uses a simple fixed sparsity pattern based on flat block butterfly and low-rank matrices to sparsify most network layers (e.g., attention, MLP). We empirically validate that Pixelated Butterfly is 3x faster than butterfly and speeds up training to achieve favorable accuracy--efficiency tradeoffs. On the ImageNet classification and WikiText-103 language modeling tasks, our sparse models train up to 2.5x faster than the dense MLP-Mixer, Vision Transformer, and GPT-2 medium with no drop in accuracy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Christopher R'e
  2. Tri Dao
  3. Atri Rudra
  4. Stefano Ermon
  5. Daniel Y. Fu
  6. Beidi Chen
  7. Nimit Sharad Sohoni
  8. Michael Poli

In Computer Vision and Pattern Recognition

  1. Christopher R'e
  2. Tri Dao
  3. Atri Rudra
  4. Stefano Ermon
  5. Daniel Y. Fu
  6. Beidi Chen
  7. Nimit Sharad Sohoni
  8. Alexander Liu

In Efficiency & Compression

  1. Tri Dao
  2. Stefano Ermon
  3. Daniel Y. Fu
  4. Christopher R'e
  5. Atri Rudra
  6. Nimit Sharad Sohoni
  7. Beidi Chen
  8. Martin Jaggi

Citations in the corpus