Self-supervised learning of a facial attribute embedding from video

By Olivia Wiles, A. Sophia Koepke, Andrew Zisserman. 2018

141 citations recorded by the source corpus.

We propose a self-supervised framework for learning facial attributes by simply watching videos of a human face speaking, laughing, and moving over time. To perform this task, we introduce a network, Facial Attributes-Net (FAb-Net), that is trained to embed multiple frames from the same video face-track into a common low-dimensional space. With this approach, we make three contributions: first, we show that the network can leverage information from multiple source frames by predicting confidence/attention masks for each frame; second, we demonstrate that using a curriculum learning regime improves the learned embedding; finally, we demonstrate that the network learns a meaningful face embedding that encodes information about head pose, facial landmarks and facial expression, i.e. facial attributes, without having been supervised with any labelled data. We are comparable or superior to state-of-the-art self-supervised methods on these tasks and approach the performance of supervised methods.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

Revisiting Self-Supervised Visual Representation Learning (2019, background): “Similarly, when learning representation from videos, one can either make use of the synchronized cross-modality stream of audio, video, and potentially subtitles [35, 39, 23, 44], or of the consistency in the temporal dimension [41].”
Unsupervised Learning of Object Keypoints for Perception and Control (2019, background): “Other works similarly either require known transformations or output dense correspondences instead of discrete landmarks [35, 29, 33, 37].”
MAST: A Memory-Augmented Self-Supervised Tracker (2020, background): “Due to the abundance of online data [1, 4, 11, 14, 15, 22, 24, 25, 26, 27, 32, 38, 43, 59, 63, 67, 68], various ideas have been explored to learn representations by exploiting the spatio-temporal information in videos.”
Learnable PINs: Cross-Modal Embeddings for Person Identity (2018, background): “Moreover, recent work has shown that the teacher networks trained for face veriﬁcation typically encode factors beyond identity such as pose [52].”
Emotion Recognition in Speech using Cross-Modal Transfer in the Wild (2018, methodology): “. Proximal supervision could also address the problem noted in Section 3, that speaking expressions can tend towards neutral. Finally, facial expressions in video can be learnt using self-supervision [62], and this offers an alternative to the strong supervision used for the teacher in this paper. 6”
Unsupervised Learning of Object Landmarks through Conditional Image Generation (2018, methodology): “[51] learn a dense deformation field for faces but obtain the template from a second related image, as in our method.”
Deepfakes generation and detection: state-of-the-art, open challenges, countermeasures, and way forward (2021, unknown): “The behavioral biometric was recognized with the encoder-decoder network (Facial Attributes-Net, FAb-Net) [206].”