Bayesian representation learning with oracle constraints

By Theofanis Karaletsos, Serge J. Belongie, Gunnar Rätsch. 2015

94 citations recorded by the source corpus.

Representation learning systems typically rely on massive amounts of labeled data in order to be trained to high accuracy. Recently, high-dimensional parametric models like neural networks have succeeded in building rich representations using either compressive, reconstructive or supervised criteria. However, the semantic structure inherent in observations is oftentimes lost in the process. Human perception excels at understanding semantics but cannot always be expressed in terms of labels. Thus, \emph{oracles} or \emph{human-in-the-loop systems}, for example crowdsourcing, are often employed to generate similarity constraints using an implicit similarity function encoded in human perception. In this work we propose to combine \emph{generative unsupervised feature learning} with a \emph{probabilistic treatment of oracle information like triplets} in order to transfer implicit privileged oracle knowledge into explicit nonlinear Bayesian latent factor models of the observations. We use a fast variational algorithm to learn the joint model and demonstrate applicability to a well-known image dataset. We show how implicit triplet information can provide rich information to learn representations that outperform previous metric learning approaches as well as generative models without this side-information in a variety of predictive tasks. In addition, we illustrate that the proposed approach compartmentalizes the latent spaces semantically which allows interpretation of the latent variables.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, background): “One can also exploit known relations between factors in different samples (Karaletsos et al., 2015; Goroshin et al., 2015; Whitney et al., 2016; Fraccaro et al., 2017; Denton and Birodkar, 2017; Hsu et al., 2017; Yingzhen and Mandt, 2018b; Locatello et al., 2018).”
Weakly-Supervised Disentanglement Without Compromises (2020, background): “For example, a number of works rely on known relations between the factors of variation (Karaletsos et al., 2015; Whitney et al., 2016; Fraccaro et al., 2017; Denton & Birodkar, 2017; Hsu et al., 2017; Yingzhen & Mandt, 2018; Locatello et al., 2018) and disentangling motion and pose from content (Hs”
On the Fairness of Disentangled Representations (2019, methodology): “Weaker forms of supervision like relational information or additional assumptions on the effect of the factors of variation were also studied [39, 16, 47, 31, 91, 26, 19, 41, 93, 62, 53, 81, 8] and applied in the sequential data and reinforcement learning settings [88, 85, 57, 69, 37, 38].”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, background): “Due to the initial poor performance of purely unsupervised methods, the field initially focused on semi-supervised [62, 11, 57, 58, 44, 46] and weakly supervised approaches [31, 12, 40, 21, 78, 20, 15, 35, 80, 54, 47, 64, 8].”
Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations (2018, background): “One can also exploit known relations between factors in different samples (Karaletsos et al., 2015; Goroshin et al., 2015; Whitney et al., 2016; Fraccaro et al., 2017; Denton & Birodkar, 2017; Hsu et al., 2017; Yingzhen & Mandt, 2018) or explicit inductive biases (Locatello et al.”
Learning Deep Disentangled Embeddings with the F-Statistic Loss (2018, background): “instances partitioned by factor value (Reed et al., 2014; Kingma et al., 2014; Kulkarni et al., 2015; Karaletsos et al., 2015; Reed et al., 2015).”
Towards a Definition of Disentangled Representations (2018, background): “However, many other approaches [50, 23, 66, 63, 80, 78, 36, 17, 77, 46] and a recent disentanglement metric by [65] consider it perfectly ﬁne for each data generative factor to be encoded by multiple latent dimensions.”
Isolating Sources of Disentanglement in VAEs (2018, background): “Many have proposed classifier-based metrics for assessing the quality of disentanglement [7, 8, 12, 13, 14, 15].”
Life-Long Disentangled Representation Learning with Cross-Domain Latent Homologies (2018, background): “A similar masking has also been used by [24] to enforce disentanglement in a single environment, but assuming additional side-information about the generative factors.”
Variational Inference of Disentangled Latent Concepts from Unlabeled Observations (2017, background): “, 2017), unless extra supervision on the generative factors is provided (Kulkarni et al., 2015; Karaletsos et al., 2015).”
Understanding disentangling in β-VAE (2018, background): “Most initial attempts to learn disentangled representations required supervised knowledge of the data generative factors [18, 35, 32, 44, 43, 12, 24, 7, 42, 20].”
DARLA: Improving Zero-Shot Transfer in Reinforcement Learning (2017, background): “…al., 2012; Cohen & Welling, 2014; 2015; Kulkarni et al., 2015; Hinton et al., 2011; Rippel & Adams, 2013; Reed et al., 2014; Yang et al., 2015; Goroshin et al., 2015; Kulkarni et al., 2015; Cheung et al., 2015; Whitney et al., 2016; Karaletsos et al., 2016; Chen et al., 2016; Higgins et al., 2017).”
Harnessing Deep Neural Networks with Logic Rules (2016, background): “…to recent neural autoencoders (Kingma and Welling, 2014; Rezende et al., 2014) where generative models encode probabilistic structures and neural recognition models distill the information through iterative optimization (Rezende et al., 2016; Johnson et al., 2016; Karaletsos et al., 2016).”
Structured Disentangled Representations (2018, methodology): “16 Table 3: Disentanglement scores (± one standard deviation) for the dSprites dataset using the metrics proposed by Kim and Mnih [2018] and Eastwood and Williams [2018].”
Disentangling Factors of Variation Using Few Labels (2019, background): “We conclude that the semi-supervised loss improves the training and transfer better to different metrics 0.84 0.86 0.88 0.90 0.92 0.94 0.96 V a lu e Metric = BetaVAE Score 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 Metric = FactorVAE Score 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 Metric = MIG 0.15 0.20 ”
Preserving Causal Constraints in Counterfactual Explanations for Machine Learning Classifiers (2019, background): “Similar ideas of capturing knowledge implicitly using oracles can be found in Karaletsos et al. [8].”
beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework (2016, background): “…a priori knowledge of the number and/or nature of the data generative factors (Hinton et al., 2011; Rippel & Adams, 2013; Reed et al., 2014; Zhu et al., 2014; Yang et al., 2015; Goroshin et al., 2015; Kulkarni et al., 2015; Cheung et al., 2015; Whitney et al., 2016; Karaletsos et al., 2016).”