Unsupervised Learning of Predictors from Unpaired Input-Output Samples
By Jianshu Chen, Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng. 2016
8 citations recorded by the source corpus.
Unsupervised learning is the most challenging problem in machine learning and especially in deep learning. Among many scenarios, we study an unsupervised learning problem of high economic value --- learning to predict without costly pairing of input data and corresponding labels. Part of the difficulty in this problem is a lack of solid evaluation measures. In this paper, we take a practical approach to grounding unsupervised learning by using the same success criterion as for supervised learning in prediction tasks but we do not require the presence of paired input-output training data. In particular, we propose an objective function that aims to make the predicted outputs fit well the structure of the output while preserving the correlation between the input and the predicted output. We experiment with a synthetic structural prediction problem and show that even with simple linear classifiers, the objective function is already highly non-convex. We further demonstrate the nature of this non-convex optimization problem as well as potential solutions. In particular, we show that with regularization via a generative model, learning with the proposed unsupervised objective function converges to an optimal solution.
Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Eric P. Xing
- Graham Neubig
- Jianfeng Gao
- Zhiting Hu
- Quoc V. Le
- Yejin Choi
- Ruslan Salakhutdinov
In Representation & Self-Supervised Learning
- Taylor Berg-Kirkpatrick
- Eric P. Xing
- Zhiting Hu
- Peter J. Liu
- Zichao Yang
- Chris Dyer
- Prajit Ramachandran
- Quoc V. Le
In Signal Processing
- Taylor Berg-Kirkpatrick
- Eric P. Xing
- Chris Dyer
- Zhiting Hu
- Quoc V. Le
- Prajit Ramachandran
- Peter J. Liu
- Zichao Yang
Citations in the corpus
- Unsupervised Pretraining for Sequence to Sequence Learning (2016, background): “Related to our approach in principle is the work by Chen et al. (2016) who proposed a two-term, theoretically motivated unsupervised objective for unpaired input-output samples.”
- Unsupervised Text Style Transfer using Language Models as Discriminators (2018, methodology): “(Liu et al., 2017; Chen et al., 2016a) use a language model as training supervision for unsupervised OCR.”
- Deep Reinforcement Learning: An Overview (2017, unknown): “Some recent papers follow: Asri et al. (2016), Bordes and Weston (2016), Chen et al. (2016b), Dhingra et al. (2016), Fatemi et al. (2016), Li et al. (2016a), Lipton et al. (2016), Mesnil et al. (2015), Mo et al. (2016), Shah et al. (2016), Su et al. (2016a), Wen et al. (2015a), Williams and Zweig…”