Tensorizing Neural Networks

By Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, Dmitry P. Vetrov. 2015

998 citations recorded by the source corpus.

Deep neural networks currently demonstrate state-of-the-art performance in several domains. At the same time, models of this class are very demanding in terms of computational resources. In particular, a large amount of memory is required by commonly used fully-connected layers, making it hard to use the models on low-end devices and stopping the further increase of the model size. In this paper we convert the dense weight matrices of the fully-connected layers to the Tensor Train [17] format such that the number of parameters is reduced by a huge factor and at the same time the expressive power of the layer is preserved. In particular, for the Very Deep VGG networks [21] we report the compression factor of the dense weight matrix of a fully-connected layer up to 200000 times leading to the compression factor of the whole network up to 7 times.

Subfields: Artificial Intelligence, Computational Mathematics, Hardware and Architecture

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Yoshua Bengio
  2. Mohit Bansal
  3. Dacheng Tao
  4. Sergey Levine
  5. Jianfeng Gao
  6. Quoc V. Le
  7. Luke S. Zettlemoyer
  8. Percy Liang

In Computational Mathematics

  1. Timothy M. Hospedales
  2. Yongxin Yang
  3. Andrzej S. Cichocki
  4. Dmitry P. Vetrov
  5. Qibin Zhao
  6. Anh Huy Phan
  7. M. A. Carreira-Perpiñán
  8. Yerlan Idelbayev

In Hardware and Architecture

  1. M. A. Carreira-Perpiñán
  2. Yerlan Idelbayev
  3. Edouard Grave
  4. Dmitry P. Vetrov
  5. Anima Anandkumar
  6. Armand Joulin
  7. Timothy M. Hospedales
  8. Victor S. Lempitsky

Citations in the corpus