Dual Supervised Learning

By Yingce Xia, Tao Qin, Jiang Bian, Nenghai Yu, Tie-Yan Liu. 2017

148 citations recorded by the source corpus.

Many supervised learning tasks are emerged in dual forms, e.g., English-to-French translation vs. French-to-English translation, speech recognition vs. text to speech, and image classification vs. image generation. Two dual tasks have intrinsic connections with each other due to the probabilistic correlation between their models. This connection is, however, not effectively utilized today, since people usually train the models of two dual tasks separately and independently. In this work, we propose training the models of two dual tasks simultaneously, and explicitly exploiting the probabilistic correlation between them to regularize the training process. For ease of reference, we call the proposed approach \emph{dual supervised learning}. We demonstrate that dual supervised learning can improve the practical performances of both tasks, for various applications including machine translation, image processing, and sentiment analysis.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Microsoft Research Asia’s Systems for WMT19 (2019, methodology): “Furthermore, some other techniques such as de-liberation learning (Xia et al., 2017b), adversarial learning (Wu et al., 2018b), and reinforcement learning (He et al., 2017; Wu et al., 2018a) could also hep and are worthy of exploration.”
Tagged Back-Translation (2019, background): “…et al., 2015, 2017), back-translation (Sennrich et al., 2016; Poncelas et al., 2018), unsupervised machine translation (Lample et al., 2018a; Artetxe et al., 2018), dual learning (Cheng et al., 2016; Di He and Ma, 2016; Xia et al., 2017), and multi-task learning (Domhan and Hieber, 2017).”
Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization (2018, methodology): “Our work differs from [14] in that we formulate the problem in an adversarial learning setup.”
Achieving Human Parity on Automatic Chinese to English News Translation (2018, methodology): “Specifically, we utilize a generic Dual Learning approach [19, 47, 46] (Section 3.”
Automatically Learning Data Augmentation Policies for Dialogue Tasks (2019, neutral): “…Lemley et al., 2017) and NLP (F¨urstenau and Lapata, 2009; Sen-nrich et al., 2016; Wang and Yang, 2015; Zhang et al., 2015; Jia and Liang, 2016; Kim and Rush, 2016; Hu et al., 2017; Xu et al., 2017; Xia et al., 2017; Silfverberg et al., 2017; Kaﬂe et al., 2017; Hou et al., 2018; Wang et al., 2018).”
Dual Transfer Learning for Neural Machine Translation with Marginal Distribution Regularization (2018, methodology): “Beyond the standard encoder-decoder architecture, more elaborate decoder architectures have been proposed to promote the performance of NMT systems (Xia et al. 2017c; He et al. 2017).”
Closed-Loop Matters: Dual Regression Networks for Single Image Super-Resolution (2020, background): “Dual learning methods [17, 40, 41, 53] contain a primal model and a dual model to learn two opposite mappings simultaneously to enhance the performance of language translation.”
Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learning (2019, unknown): “(3) Similar ways of decomposition can also be found in dual learning (Xia et al., 2017) on machine translation.”
Question Answering and Question Generation as Dual Tasks (2017, methodology): “Our approach differs from Generative Domain-Adaptive Nets (Yang et al. 2017) in that we do not pretrain the QA model.”
Semantic Parsing with Dual Learning (2019, methodology): “The idea of dual learning has been applied in various tasks (Xia et al., 2017), such as Question Answering/Generation (Tang et al., 2017, 2018), Image-to-Image Translation (Yi et al., 2017) and Open-domain Information Extraction/Narration (Sun et al., 2018).”
Deliberation Networks: Sequence Generation Beyond One-Pass Decoding (2017, methodology): “Furthermore, we leverage our recently proposed dual learning techniques [8, 33] to improve the model, which takes another 7 days.”
Reconstruction Network for Video Captioning (2018, background): “As far as we know, dual learning mechanism has not been employed in video captioning but widely used in NMT [37, 11, 45].”
APE at Scale and Its Implications on MT Evaluation Biases (2019, background): “…(Brants et al., 2007) and neural MT (G¨ulc¸ehre et al., 2015, 2017), back-translation (Sennrich et al., 2016b), unsupervised NMT (Lample et al., 2018a; Artetxe et al., 2018a), dual learning (Cheng et al., 2016; He et al., 2016; Xia et al., 2017), and multi-task learning (Domhan and Hieber, 2017).”
Neural Machine Translation: A Review (2019, background): “The dual supervised learning loss LDSL aims to correlate both translation directions as follows (Hassan et al., 2018; Xia et al., 2017):”
Decoding with Value Networks for Neural Machine Translation (2017, background): “While there are different objectives to achieve this [2, 10, 1, 5, 19, 17], maximum likelihood estimation is the most popular one [2]: Θ∗ = argmax Θ ∏”
Multi-Agent Dual Learning (2019, methodology): “…synthesis, dual learning is proposed (He et al., 2016) and applied to many applications including neural machine translation (NMT) (He et al., 2016; Xia et al., 2017a;b), image-to-image translation (Zhu et al., 2017; Yi et al., 2017; Luo et al., 2017), question answering (Duan et al., 2017; Sun et…”
Learning Dual Retrieval Module for Semi-supervised Relation Extraction (2019, background): “Our work is also related to dual learning [14, 32, 35], which can be generalized to the concept of jointly learning the primal and dual models to boost the performance of each other.”