Trunk-Branch Ensemble Convolutional Neural Networks for Video-Based Face Recognition

By Changxing Ding, Dacheng Tao. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016

406 citations recorded by the source corpus.

Human faces in surveillance videos often suffer from severe image blur, dramatic pose variations, and occlusion. In this paper, we propose a comprehensive framework based on Convolutional Neural Networks (CNN) to overcome challenges in video-based face recognition (VFR). First, to learn blur-robust face representations, we artificially blur training data composed of clear still images to account for a shortfall in real-world video training data. Using training data composed of both still images and artificially blurred data, CNN is encouraged to learn blur-insensitive features automatically. Second, to enhance robustness of CNN features to pose variations and occlusion, we propose a Trunk-Branch Ensemble CNN model (TBE-CNN), which extracts complementary information from holistic face images and patches cropped around facial components. TBE-CNN is an end-to-end model that extracts features efficiently by sharing the low- and middle-level convolutional layers between the trunk and branch networks. Third, to further promote the discriminative power of the representations learnt by TBE-CNN, we propose an improved triplet loss function. Systematic experiments justify the effectiveness of the proposed techniques. Most impressively, TBE-CNN achieves state-of-the-art performance on three popular video face databases: PaSC, COX Face, and YouTube Faces. With the proposed techniques, we also obtain the first place in the BTAS 2016 Video Person Recognition Evaluation.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/16f31b3d11d11c08f81d0164d72a53191a7f904b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Xiaogang Wang
4. Ming-Hsuan Yang
5. Xiaoou Tang
6. Jie Zhou
7. Jiwen Lu
8. Weihong Deng

In Signal Processing

1. Mei Wang
2. Weihong Deng
3. Jie Zhou
4. Jiwen Lu
5. Fang Wen
6. Dong Chen
7. Yoanna Martínez-Díaz
8. Leyanis López-Avila

Citations in the corpus

• Neural Aggregation Network for Video Face Recognition (2016, methodology): “Maxor average-pooling is used to aggregate the frame features in [25, 11, 7, 8].”
• Toward More Realistic Face Recognition Evaluation Protocols for the YouTube Faces Database (2018, methodology): “Currently, the best-performing methods on the YTF database report more than 90% of accuracy by using deep learning-based representations [4, 5, 13, 16].”
• Attention-Aware Deep Reinforcement Learning for Video Face Recognition (2017, methodology): “To further show the effectiveness of our method, we have conducted additional experiments by fine-tuning CNN model following [10], which we referred to as ADRLfinetune.”
• Examining the Impact of Blur on Recognition by Convolutional Networks (2016, background): “trained for face recognition [10, 17, 4], while Ullman et al.”
• Deep Face Recognition: A Survey (2018, background): “[47] artificially blurred clear still images for training to learn blur-robust face representations.”