General Facial Representation Learning in a Visual-Linguistic Manner

By Yinglin Zheng, Hao Yang, Ting Zhang, Jianmin Bao, Dongdong Chen, Yangyu Huang, Lu Yuan, Ming Zeng, Fang Wen. 2021

269 citations recorded by the source corpus.

How to learn a universal facial representation that boosts all face analysis tasks? This paper takes one step toward this goal. In this paper, we study the transfer performance of pre-trained models on face analysis tasks and introduce a framework, called FaRL, for general facial representation learning. On one hand, the framework involves a contrastive loss to learn high-level semantic meaning from image-text pairs. On the other hand, we propose exploring low-level information simultaneously to further enhance the face representation by adding a masked image modeling. We perform pre-training on LAION-FACE, a dataset containing a large amount of face image-text pairs, and evaluate the representation capability on multiple downstream tasks. We show that FaRL achieves better transfer performance compared with previous pre-trained models. We also verify its superiority in the low-data regime. More importantly, our model surpasses the state-of-the-art methods on face analysis tasks including face parsing and face alignment.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neurology, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Lu Yuan
2. Dongdong Chen
3. Dong Chen
4. Nenghai Yu
5. Fang Wen
6. Ting Zhang
7. Weiming Zhang
8. Xiaoyi Dong

In Computer Vision and Pattern Recognition

1. Lu Yuan
2. Xiaogang Wang
3. Luc Van Gool
4. Dongdong Chen
5. Fang Wen
6. Dong Chen
7. Xiaoou Tang
8. Ming-Hsuan Yang

In Neurology

1. Lu Yuan
2. Nenghai Yu
3. Dongdong Chen

In Multimodal Learning

1. Lu Yuan
2. Fang Wen
3. Ting Zhang
4. Dong Chen
5. Dongdong Chen
6. Jianmin Bao

In Representation & Self-Supervised Learning

1. Lu Yuan
2. Nenghai Yu
3. Fang Wen
4. Ting Zhang
5. Dong Chen
6. Weiming Zhang
7. Dongdong Chen
8. Jianmin Bao

Citations in the corpus

• PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding (2023, unknown): “However, existing human datasets either do not annotate ID information [31, 37, 61, 78], or the richness of the scenes they contain is very limited [38, 46, 68] ( i.e ., they only focus on the face area).”
• InstantID: Zero-shot Identity-Preserving Generation in Seconds (2024, methodology): “We implement with Stable Diffusion and train it on the large-scale open-source dataset LAION-Face [27], which consists of 50 million image-text pairs to ensure diversity.”
• PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers (2021, background): “Most recently, numerous works (Bao, Dong, and Wei 2021; He et al. 2021; Xie et al. 2021; Chen et al. 2022; Wang et al. 2022b; Dong et al. 2022; Baevski et al. 2022; Zheng et al. 2022) based on MIM have been concurrently developed, yet few studied the perceptual level of the prediction targets.”