Jointly Optimize Data Augmentation and Network Training: Adversarial Data Augmentation in Human Pose Estimation

By Xi Peng, Zhiqiang Tang, Fei Yang, Rogério Schmidt Feris, Dimitris N. Metaxas. 2018

222 citations recorded by the source corpus.

Random data augmentation is a critical technique to avoid overfitting in training deep models. Yet, data augmentation and network training are often two isolated processes in most settings, yielding to a suboptimal training. Why not jointly optimize the two? We propose adversarial data augmentation to address this limitation. The key idea is to design a generator (e.g. an augmentation network) that competes against a discriminator (e.g. a target network) by generating hard examples online. The generator explores weaknesses of the discriminator, while the discriminator learns from hard augmentations to achieve better performance. A reward/penalty strategy is also proposed for efficient joint training. We investigate human pose estimation and carry out comprehensive ablation studies to validate our method. The results prove that our method can effectively improve state-of-the-art models without additional data effort.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Adversarial AutoAugment (2019, background): “Peng et al. (2018) also take such an idea to optimize the training process of a target network in human pose estimation.”
Greedy AutoAugment (2019, background): “A third example is to help generators and discriminators for training generative adversarial networks [10], [11].”
Deep High-Resolution Representation Learning for Human Pose Estimation (2019, background): “Nowadays, deep convolutional neural network provides dominant solutions [20, 34, 60, 41, 42, 47, 56, 16].”
A Generative Adversarial Approach for Zero-Shot Learning from Noisy Texts (2017, background): “Generative adversarial networks(GANs) [14] have shown promising performance on generating realistic images [47, 52, 53, 45, 30].”
Fast Human Pose Estimation (2018, background): “This boost L SP M P II is bigger than other state-of-the-art gains, e.g. +0.3% in 91.2% [21] vs 90.9% [19]; further +0.3% in 91.5% [23].”
Densely Connected Pyramid Dehazing Network (2018, background): “The success of GANs in synthesizing realistic images has led researchers to explore the adversarial loss for various low-level vision applications such as text-to-image synthesis[32, 52, 55, 6], imageimage translation [18, 28, 46, 45, 50], super-resolution [23], human pose estimation [31] and other ”
Learning Data Manipulation for Augmentation and Weighting (2019, methodology): “Recent work has developed automated augmentation approaches [5, 38, 28, 37, 47].”
A Unified Query-based Generative Model for Question Generation and Question Answering (2017, methodology): “For the future work, we will consider adding adversarial data, which has been shown successful on plenty of areas(Peng et al., 2018).”
Monocular human pose estimation: A survey of deep learning-based methods (2020, methodology): “…PRM (Yang et al., 2017), feed forward module (Belagiannis and Zisserman, 2017), HRNet (Sun et al., 2019), GAN (Chou et al., 2017; Chen et al., 2017; Peng et al., 2018) (3) Body structure constraint : (Tompson et al., 2014; Lifshitz et al., 2016; Yang et al., 2016; Gkioxari et al., 2016; Chu et…”
Bowtie Networks: Generative Modeling for Joint Few-Shot Recognition and Novel-View Synthesis (2020, background): “Joint Data Augmentation and Task Model Learning leverage generative networks to improve other visual tasks (Peng et al., 2018; Hu et al., 2019; Luo et al., 2020; Zhang et al., 2020).”
Semantic Graph Convolutional Networks for 3D Human Pose Regression (2019, methodology): “Other methods use synthetic datasets which are generated from deforming a human template model with the ground truth [8, 42, 48] or introduce loss functions involving high-level knowledge [40, 53, 69] in addition to joints.”
Adaptive Wing Loss for Robust Face Alignment via Heatmap Regression (2019, methodology): “al dataset but not in LSP extended training dataset. During training, we did not calculate loss on occluded keypoints for LSP extended training dataset. During training and testing, we did not follow [43] to crop single person from images with multiple persons to retain the difﬁculties of this datas”
Deep Learning-based Human Pose Estimation: A Survey (2020, methodology): “[189] developed an adversarial data augmentation network to optimize data augmentation and network training by treating the HPE network as a discriminator and using augmentation network as a generator to perform adversarial augmentations.”
ResizeMix: Mixing Data with Preserved Object Information and True Labels (2020, background): “Integrated with adversarial training [2, 21, 40], AdvAA [54] makes networks learn more hard data samples, in which the domain of dataset becomes more widespread.”
See Better Before Looking Closer: Weakly Supervised Data Augmentation Network for Fine-Grained Visual Classification (2019, methodology): “Peng et al. proposed Adversarial Data Augmentation [20] to jointly optimize data augmentation and deep model.”
Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression (2021, methodology): “The convolutional neural network (CNN) solutions [17, 35, 56, 70, 43, 45, 49, 74, 54] to human pose estimation have shown superior performance over the conventional methods, such as the probabilistic graphical model or the pictorial structure model [72, 50].”
Learning to Learn Single Domain Generalization (2020, methodology): “Inspired by the recent success of adversarial training [35, 50, 49, 36, 24], we cast the single domain generalization problem in a worst-case formulation [44, 20].”