Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA

By Aapo Hyvärinen, Hiroshi Morioka. 2016

504 citations recorded by the source corpus.

Nonlinear independent component analysis (ICA) provides an appealing framework for unsupervised feature learning, but the models proposed so far are not identifiable. Here, we first propose a new intuitive principle of unsupervised deep learning from time series which uses the nonstationary structure of the data. Our learning principle, time-contrastive learning (TCL), finds a representation which allows optimal discrimination of time segments (windows). Surprisingly, we show how TCL can be related to a nonlinear ICA model, when ICA is redefined to include temporal nonstationarities. In particular, we show that TCL combined with linear ICA estimates the nonlinear ICA model up to point-wise transformations of the sources, and this solution is unique --- thus providing the first identifiability result for nonlinear ICA which is rigorous, constructive, as well as very general.

Subfields: Analytical Chemistry, Artificial Intelligence, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Analytical Chemistry

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, background): “ems, in which the ‘predicted’ representation must be recognized amongst a set of distracting ‘negative’ examples. For example, camera motion [1,32], tracking [61], text proximity [41], and other cues [11,62,30,60,29] can be used to ﬁnd examples of features that should be ‘similar’ or predictable fro”
A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, methodology): “Similarly, the non-linear ICA community recently shifted to non-iid data types exploiting time dependent or grouped observations (Hyvarinen and Morioka, 2016; Hyvarinen et al., 2019; Gresele et al., 2019) We focus our study on the setting where factors of variations are not observable at all, that i”
Representation Learning with Contrastive Predictive Coding (2018, methodology): “In Time Contrastive Learning [25] a contrastive loss is used to predict the segment-ID of multivariate time-series as a way to extract features and perform nonlinear ICA.”
Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA (2021, methodology): “Recent theoretical results have shown that nonlinear ICA is possible when leveraging additional assumptions, e.g., nonstationarity (Hyvarinen and Morioka, 2016) and temporal dependencies (Hyvarinen and Morioka, 2017).”
Unsupervised Pretraining Transfers Well Across Languages (2020, background): “In particular, Time Contrastive Learning [5] a r X i v : 2002 .”
Colorization as a Proxy Task for Visual Understanding (2017, background): “Recent follow-up work generalizes this to a nonlinear setting [11].”
On the Transfer of Disentangled Representations in Realistic Settings (2020, background): “Ideas related to disentangling the factors of variation date back to the non-linear ICA literature (Comon, 1994; Hyv¨arinen and Pajunen, 1999; Bach and Jordan, 2002; Jutten and Karhunen, 2003; Hyvarinen and Morioka, 2016; Hyvarinen et al., 2019; Gresele et al., 2019).”
Hidden Markov Nonlinear ICA: Unsupervised Learning from Nonstationary Time Series (2020, methodology): “Recent theoretical and practical advances in Nonlinear ICA provide a principled approach to this problem (Hyv¨arinen and Morioka, 2016; Hyv¨arinen and Morioka, 2017; Hyv¨arinen et al., 2019; Khemakhem et al., 2020; Sorrenson et al., 2020).”
Weakly-Supervised Disentanglement Without Compromises (2020, background): “Recently, the ICA community has considered weak forms of supervision such as temporal consistency (Hyvarinen & Morioka, 2016; 2017), auxiliary supervised information (Hyvarinen et al., 2019; Khemakhem et al., 2019), and multiple views (Gresele et al., 2019).”
On the Fairness of Disentangled Representations (2019, background): “Ideas related to disentangling the factors of variations have a long tradition in machine learning, dating back to the non-linear ICA literature [17, 3, 46, 42, 43, 44, 32].”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, background): “Disentangling pose, style, or motion from content are classical vision tasks that has been studied with different degrees of supervision [72, 79, 80, 34, 19, 14, 21, 36].”
The Incomplete Rosetta Stone problem: Identifiability results for Multi-view Nonlinear ICA (2019, background): “The setup in [19, 20, 23] makes strong assumptions on the data generating mechanism, but allows for arbitrary nonlinear mixing of the sources.”
Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations (2018, methodology): “The lack of almost any identifiability result in nonlinear ICA has been a main bottleneck for the utility of the approach (Hyvarinen et al., 2018) and partially motivated alternative machine learning approaches (Desjardins et al., 2012; Schmidhuber, 1992; Cohen & Welling, 2015).”
Learning Independent Causal Mechanisms (2017, background): “On the conceptual level, our setting is related to recent work on deep learning for disentangling factors of variation (Chen et al., 2016; Higgins et al., 2017) as well as non-linear ICA (Hyvarinen & Morioka, 2016).”
Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning (2018, methodology): “This unifies and generalizes the methods in [11, 21, 15, 14].”
Variational Autoencoders and Nonlinear ICA: A Unifying Framework (2019, methodology): “Dataset We run simulations on data used previously in the nonlinear ICA literature (Hyvärinen and Morioka, 2016; Hyvärinen et al., 2019).”
Disentangling the independently controllable factors of variation by interacting with the world (2018, background): “…[Kingma and Welling, 2014] , generative adversarial networks [Goodfellow et al., 2014] or non-linear versions of ICA [Dinh et al., 2014, Hyvarinen and Morioka, 2016] attempt to disentangle the underlying factors of variation by assuming that their joint distribution (marginalizing out…”
Disentangling Factors of Variation Using Few Labels (2019, background): “There is for example a rich literature in disentangling pose from content in 3D objects and content from motion in videos [64, 65, 24, 15, 12, 18, 26].”
Deep Energy Estimator Networks (2018, methodology): “This problem was recently reformulated for time series, solving the nonidentiﬁability problem (Hyvärinen & Morioka, 2016).”