Communication-Optimal Convolutional Neural Nets

By J. Demmel, Grace Dinh. 2018

22 citations recorded by the source corpus.

Efficiently executing convolutional neural nets (CNNs) is important in many machine-learning tasks. Since the cost of moving a word of data, either between levels of a memory hierarchy or between processors over a network, is much higher than the cost of an arithmetic operation, minimizing data movement is critical to performance optimization. In this paper, we present both new lower bounds on data movement needed for CNNs, and optimal sequential algorithms that attain these lower bounds. In most common cases, our optimal algorithms can attain significantly more data reuse than matrix multiplication.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Ashish Vaswani
2. Quoc V. Le
3. Niki Parmar
4. Noam Shazeer
5. Jianfeng Gao
6. Dustin Tran
7. Irwan Bello
8. Mohit Bansal

In Computational Mechanics

1. Noam Shazeer
2. Niki Parmar
3. Ashish Vaswani
4. Dustin Tran

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Noam Shazeer
3. Quoc V. Le
4. Dacheng Tao
5. Yi Tay
6. Andrew Zisserman
7. Trevor Darrell
8. Xiaogang Wang

In Optimization & Training

1. HyoukJoong Lee
2. Ashish Vaswani
3. Blake A. Hechtman
4. Noam Shazeer
5. Peter G. Hawkins
6. Mingsheng Hong
7. Penporn Koanantakool
8. Ryan Sepassi

Citations in the corpus

• Mesh-TensorFlow: Deep Learning for Supercomputers (2018, methodology): “5D [14] algorithm for matrices with different sparsities, best tile sizes for direct convolutions [17], etc.”