Factorized Higher-Order CNNs With an Application to Spatio-Temporal Emotion Estimation

By Jean Kossaifi, Antoine Toisoul, Adrian Bulat, Yannis Panagakis, Timothy M. Hospedales, Maja Pantic. 2019

89 citations recorded by the source corpus.

Training deep neural networks with spatio-temporal (i.e., 3D) or multidimensional convolutions of higher-order is computationally challenging due to millions of unknown parameters across dozens of layers. To alleviate this, one approach is to apply low-rank tensor decompositions to convolution kernels in order to compress the network and reduce its number of parameters. Alternatively, new convolutional blocks, such as MobileNet, can be directly designed for efficiency. In this paper, we unify these two approaches by proposing a tensor factorization framework for efficient multidimensional (separable) convolutions of higher-order. Interestingly, the proposed framework enables a novel higher-order transduction, allowing to train a network on a given domain (e.g., 2D images or N-dimensional data in general) and using transduction to generalize to higher-order data such as videos (or (N+K)--dimensional data in general), capturing for instance temporal dynamics while preserving the learnt spatial information. We apply the proposed methodology, coined CP-Higher-Order Convolution (HO-CPConv), to spatio-temporal facial emotion analysis. Most existing facial affect models focus on static imagery and discard all temporal information. This is due to the above-mentioned burden of training 3D convolutional nets and the lack of large bodies of video data annotated by experts. We address both issues with our proposed framework. Initial training is first done on static imagery before using transduction to generalize to the temporal domain. We demonstrate superior performance on three challenging large scale affect estimation datasets, AffectNet, SEWA, and AFEW-VA.

Subfields: Computational Mathematics, Computer Vision and Pattern Recognition, Experimental and Cognitive Psychology, Model Architectures, Sentiment & Opinion Analysis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mathematics

1. Nuno Vasconcelos

In Computer Vision and Pattern Recognition

1. Lu Yuan
2. Luc Van Gool
3. Xiyang Dai
4. Jianfeng Gao
5. Zicheng Liu
6. Dacheng Tao
7. Dongdong Chen
8. Jianwei Yang

In Model Architectures

1. Lu Yuan
2. Xiyang Dai
3. Zicheng Liu
4. Dongdong Chen
5. Yinpeng Chen
6. Mengchen Liu
7. Nuno Vasconcelos
8. Yunsheng Li

Citations in the corpus

• Revisiting Dynamic Convolution via Matrix Decomposition (2021, methodology): “More recently, Kossaifi et al. (2020) combines tensor decompositions with MobileNet to design efficient higher-order networks for video tasks, while Phan et al. (2020) proposes a stable CPD to deal with degeneracies of tensor decompositions during network training.”