Multimodal Distributional Semantics

By Elia Bruni, Nam Khanh Tran, Marco Baroni. 2013

937 citations recorded by the source corpus.

Distributional semantic models derive computational representations of word meaning from the patterns of co-occurrence of words in text. Such models have been a success story of computational linguistics, being able to provide reliable estimates of semantic relatedness for the many semantic tasks requiring them. However, distributional models extract meaning information exclusively from text, which is an extremely impoverished basis compared to the rich perceptual sources that ground human semantic knowledge. We address the lack of perceptual grounding of distributional models by exploiting computer vision techniques that automatically identify discrete "visual words" in images, so that the distributional representation of a word can be extended to also encompass its co-occurrence with the visual words of images it is associated with. We propose a flexible architecture to integrate text- and image-based distributional information, and we show in a set of empirical tests that our integrated model is superior to the purely text-based approach, and it provides somewhat complementary semantic information with respect to the latter.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Using the Output Embedding to Improve Language Models (2016, methodology): “We use the data as processed by (Cho et al., 2014) using the data selection method of (Axelrod et al., 2011).”
Experience Grounds Language (2020, background): “Importantly, this is not a new trend (Chen and Mooney, 2008; Feng and La-pata, 2010; Bruni et al., 2014; Lazaridou et al., 2016), and task and model design can fail to require sensory perception (Thomason et al., 2019a).”
ConceptNet 5.5: An Open Multilingual Graph of General Knowledge (2016, methodology): “Its high scores on both the Rare Words dataset and the crowd-sourced MEN-3000 and MTurk-771 datasets, exceeding the performance of other embeddings with high conﬁdence, shows both the breadth and the depth of its understanding of words.”
Fusion of Detected Objects in Text for Visual Question Answering (2019, background): “However, the same capacity allows for multimodal context, which may help model the meaning of words in general, and also sharpen its understanding of instances of words in context (e.g., Bruni et al. 2014).”
Bayesian Neural Word Embedding (2016, methodology): “The word similarity evaluation includes several different datasets: WordSim353 [16], SCWS [19], Rare Words (RW) [20], MEN [21] and Sim-Lex999 [22].”
Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors (2014, background): “Bruni et al. (2013), the developers of this benchmark, achieve state-of-the-art performance by extensive tuning on ad-hoc training data, and by using both textual and image-extracted features to represent word meaning.”
Supervised Multimodal Bitransformers for Classifying Images and Text (2019, background): “for lexical representation learning [7, 25].”
Massively Multilingual Word Embeddings (2016, neutral): “Word similarity datasets such as WordSim-353 (Agirre et al., 2009) and MEN (Bruni et al., 2014) provide human judgments of semantic similarity.”
Jointly optimizing word representations for lexical and sentential tasks with the C-PHRASE model (2015, methodology): “We used, ﬁrst of all, the MEN ( men ) data set of Bruni et al. (2014), that is split into 1K pairs for training/development, and 1K pairs for testing.”
How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings (2019, background): “These tasks cover semantic similarity, analogy solving, and concept categorization: SimLex999 (Hill et al., 2015), MEN (Bruni et al., 2014), WS353 (Finkelstein et al., 2002), RW (Luong et al., 2013), SemEval-2012 (Jurgens et al., 2012), Google analogy solving (Mikolov et al., 2013a) MSR analogy solv”
All-but-the-Top: Simple and Effective Postprocessing for Word Representations (2017, methodology): “…353 pairs of commonly used verbs and nouns; the rare-words (RW) dataset (Lu-ong et al., 2013) composed of rarely used words; the MEN dataset (Bruni et al., 2014) where the 3000 pairs of words are rated by crowdsourced participants; the MTurk dataset (Radinsky et al., 2011) where the 287…”
Rethinking embedding coupling in pre-trained language models (2020, methodology): “We evaluate on MEN (Bruni et al., 2014), MTurk771 (Halawi et al.”
Combining Language and Vision with a Multimodal Skip-gram Model (2015, methodology): “MDSMs outperform state-of-the-art text-based approaches, not only in tasks that directly require access to visual knowledge (Bruni et al., 2012), but also on general semantic benchmarks (Bruni et al., 2014; Silberer and Lapata, 2014).”
Evaluation methods for unsupervised word embeddings (2015, methodology): “The more recent MEN dataset (Bruni et al., 2014) follows a similar strategy, but restricts queries to words that occur as annotations in an image dataset.”
Semantic Speech Retrieval With a Visually Grounded Model of Untranscribed Speech (2017, background): “…so most studies evaluate models using soft human ratings on tasks such as word similarity (Feng and Lapata, 2010; Sil-berer and Lapata, 2012; Bruni et al., 2014), word association (Bruni et al., 2012; Roller and Schulte Im Walde, 2013) or concept categorisation (Silberer and Lapata, 2014).”
Supersense Embeddings: A Unified Model for Supersense Interpretation, Prediction, and Utilization (2016, neutral): “R) (Agirre et al., 2009), MEN dataset (Bruni et al., 2014), RG-65 dataset (Rubenstein and Goodenough, 1965) and MC-30 (Miller and Charles, 1991).”
Embedding Words and Senses Together via Joint Knowledge-Enhanced Training (2016, methodology): “SimLex and MEN are composed by 999 and 3000 word pairs, respectively.”