Tensor Regression Networks

By Jean Kossaifi, Zachary Chase Lipton, Aran Khanna, Tommaso Furlanello, Anima Anandkumar. 2017

173 citations recorded by the source corpus.

Convolutional neural networks typically consist of many convolutional layers followed by one or more fully connected layers. While convolutional layers map between high-order activation tensors, the fully connected layers operate on flattened activation vectors. Despite empirical success, this approach has notable drawbacks. Flattening followed by fully connected layers discards multilinear structure in the activations and requires many parameters. We address these problems by incorporating tensor algebraic operations that preserve multilinear structure at every layer. First, we introduce Tensor Contraction Layers (TCLs) that reduce the dimensionality of their input while preserving their multilinear structure using tensor contraction. Next, we introduce Tensor Regression Layers (TRLs), which express outputs through a low-rank multilinear mapping from a high-order activation tensor to an output tensor of arbitrary order. We learn the contraction and regression factors end-to-end, and produce accurate nets with fewer parameters. Additionally, our layers regularize networks by imposing low-rank constraints on the activations (TCL) and regression weights (TRL). Experiments on ImageNet show that, applied to VGG and ResNet architectures, TCLs and TRLs reduce the number of parameters compared to fully connected layers by more than 65% while maintaining or increasing accuracy. In addition to the space savings, our approach's ability to leverage topological structure can be crucial for structured data such as MRI. In particular, we demonstrate significant performance improvements over comparable architectures on three tasks associated with the UK Biobank dataset.

Subfields: Computational Mathematics, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mathematics

1. Jean Kossaifi
2. U. Kang
3. Minyong Cho
4. Taebum Kim
5. Jaemin Yoo
6. Yuan Xie
7. Song Han
8. Maja Pantic

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Xiaogang Wang
4. Timothy M. Hospedales
5. Yi Yang
6. Chen Change Loy
7. Trevor Darrell
8. Ming-Hsuan Yang

In Radiology, Nuclear Medicine and Imaging

1. Song Han
2. Junier B. Oliva
3. Adrian Bulat
4. Maja Pantic
5. Timothy M. Hospedales
6. Yannis Panagakis
7. Eunbyung Park
8. Jean Kossaifi

In Model Architectures

1. Yuan Xie
2. Song Han
3. Junier B. Oliva
4. Adrian Bulat
5. Maja Pantic
6. By Lei Deng
7. Timothy M. Hospedales
8. U. Kang

Citations in the corpus

• Meta-Curvature (2019, background): “[18] proposed to learn the projection factors in Tucker decomposition for fully connected layers in deep networks.”
• Factorized Higher-Order CNNs With an Application to Spatio-Temporal Emotion Estimation (2019, methodology): “For our factorized higher-order convolutional network, we further removed the flattening and fully-connected layers and replace them with a single tensor regression layer [21] in order to fully preserve the structure in the activations.”
• Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, methodology): “Instead of directly replacing weights with decomposed tensors, Chien and Bao [101] considered the entire decomposition process of (19) as a connection, similar to [102] and [103].”
• Knowledge Extraction with No Observable Data (2019, methodology): “[17] used Tucker decomposition to compress fully connected layers as well as convolutional layers.”