Representation Learning with Contrastive Predictive Coding

By A. van den Oord, Aäron van den Oord, Yazhe Li, Oriol Vinyals. 2018

13,907 citations recorded by the source corpus.

While supervised learning has enabled great progress in many applications, unsupervised learning has not seen such widespread adoption, and remains an important and challenging endeavor for artificial intelligence. In this work, we propose a universal unsupervised learning approach to extract useful representations from high-dimensional data, which we call Contrastive Predictive Coding. The key insight of our model is to learn such representations by predicting the future in latent space by using powerful autoregressive models. We use a probabilistic contrastive loss which induces the latent space to capture information that is maximally useful to predict future samples. It also makes the model tractable by using negative sampling. While most prior work has focused on evaluating representations for a particular modality, we demonstrate that our approach is able to learn useful representations achieving strong performance on four distinct domains: speech, images, text and reinforcement learning in 3D environments.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Yoshua Bengio
  3. Jianfeng Gao
  4. Sergey Levine
  5. Quoc V. Le
  6. Percy Liang
  7. Ruslan Salakhutdinov
  8. Luke S. Zettlemoyer

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Ming-Hsuan Yang
  5. Xiaogang Wang
  6. Jitendra Malik
  7. Andrew Zisserman
  8. Andrea Vedaldi

In Representation & Self-Supervised Learning

  1. Yoshua Bengio
  2. Trevor Darrell
  3. Ishan Misra
  4. Andrea Vedaldi
  5. Andrew Zisserman
  6. Dacheng Tao
  7. Sergey Levine
  8. Yann LeCun

Citations in the corpus