Learning Grounded Meaning Representations with Autoencoders

By Carina Silberer, Mirella Lapata. 2014

229 citations recorded by the source corpus.

In this paper we address the problem of grounding distributional representations of lexical meaning. We introduce a new model which uses stacked autoencoders to learn higher-level embeddings from textual and visual input. The two modalities are encoded as vectors of attributes and are obtained automatically from text and images, respectively. We evaluate our model on its ability to simulate similarity judgments and concept categorization. On both tasks, our approach outperforms baselines and related models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Experience Grounds Language (2020, background): “Most of these works provide resources such as data, code, simulators and methodology for evaluating the multimodal content of linguistic representations (Schlangen, 2019b; Silberer and Lapata, 2014; Bruni et al., 2012).”
SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation (2014, methodology): “As in previous annotation or data collection for computational semantics (Pad´o et al., 2007; Reisinger and Mooney, 2010a; Silberer and Lapata, 2014) we computed the inter-rater agreement as the average of pairwise Spearman ρ correlations between the ratings of all respondents.”
Combining Language and Vision with a Multimodal Skip-gram Model (2015, methodology): “MDSMs outperform state-of-the-art text-based approaches, not only in tasks that directly require access to visual knowledge (Bruni et al., 2012), but also on general semantic benchmarks (Bruni et al., 2014; Silberer and Lapata, 2014).”
COALA: Co-Aligned Autoencoders for Learning Semantically Enriched Audio Representations (2020, methodology): “Similar approaches have been used in (Silberer & Lapata, 2014), where an AE structure was employed in order to learn an embedding from a k-hot encoding of tags/words that would encapsulate semantic information.”
Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models (2019, background): “A d a p t a t i o n RBM-CCA (Ngiam et al., 2011) x x SBA (Silberer and Lapata, 2014) – – – GRL (Ganin and Lempitsky, 2015) – – – – DAN (Long et al., 2015) – – – – DSN (Bousmalis et al., 2016) x x – DMAE (Mukherjee et al., 2017) x x x VA E B a s e d TELBO (Vedantam et al., 2018) x x x JMVAE (Suzuki e”
Multimodal Intelligence: Representation Learning, Information Fusion, and Applications (2019, methodology): “inputs using multiple streams of deep Boltzmann machines or auto-encoders with shared layers acting as a shared representation space [107]–[109].”
A Survey of Multi-View Representation Learning (2016, unknown): “Consequently, Silberer and Lapata [174] train stacked multimodal autoencoder with semi-supervised objective to learn grounded meaning representations.”
SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity (2016, background): “IAA-1 (pairwise) computes the average pairwise Spearman’s ρ correlation between any two raters – a common choice in previous data collection in distributional semantics (Padó et al., 2007; Reisinger and Mooney, 2010a; Silberer and Lapata, 2014; Hill et al., 2015).”
Multilingual Visual Sentiment Concept Matching (2016, unknown): “visual similarity to evaluate word embeddings [13, 14].”
Deep Multimodal Representation Learning: A Survey (2019, result): “Similar to the work fromNgiam, Silberer and Lapata [105] proposed a variant to learn semantic representations from textual and visual input.”
Visually Grounded Compound PCFGs (2020, background): “It has been shown effective in word representation learning (Bruni et al., 2014; Silberer and Lapata, 2014; Lazaridou et al., 2015) and sentence representation learning (Kiela et al., 2018; Bordes et al., 2019).”
Training and Evaluating Multimodal Word Embeddings with Large-scale Web Annotated Images (2016, methodology): “One type of methods takes a two-step strategy that ﬁrst extracts text and image features separately and then fuses them together using singular value decomposition [5], stacked autoencoders [31], or even simple concatenation [17].”
Associative Multichannel Autoencoder for Multimodal Word Representation (2018, methodology): “…deep learning methods to project different modality inputs into a common space, including restricted Boltzman machines (Ngiam et al., 2011; Srivastava and Salakhutdinov, 2012), autoencoders (Silberer and Lapata, 2014; Silberer et al., 2016), and recursive neural networks (Socher et al., 2013).”
The Emergence of Semantics in Neural Network Representations of Visual Information (2018, background): “Indeed, joint models of semantics based on images and text have been developed, and word vectors have been grounded in the visual space (Silberer and Lapata, 2012, 2014; Bruni and Baroni, 2013).”
Probing Contextual Language Models for Common Ground with Visual Representations (2020, background): “Similarly, (Silberer and Lapata, 2014) builds multi-modal representations by using stacked autoencoders.”
Multimodal Machine Learning: A Survey and Taxonomy (2017, background): “Similarly, Silberer and Lapata [191] proposed to use a”