A Compositional and Interpretable Semantic Space

By Alona Fyshe, Leila Wehbe, Partha P. Talukdar, Brian Murphy, Tom Michael Mitchell. 2015

70 citations recorded by the source corpus.

Vector Space Models (VSMs) of Semantics are useful tools for exploring the semantics of single words, and the composition of words to make phrasal meaning. While many methods can estimate the meaning (i.e. vector) of a phrase, few do so in an interpretable way. We introduce a new method (CNNSE) that allows word and phrase vectors to adapt to the notion of composition. Our method learns a VSM that is both tailored to support a chosen semantic composition operation, and whose resulting features have an intuitive interpretation. Interpretability allows for the exploration of phrasal semantics, which we leverage to analyze performance on a behavioral task.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Analysis Methods in Neural Language Processing: A Survey (2018, background): “Several studies conducted behavioral experiments to interpret word embeddings by deﬁning intrusion tasks, where humans need to identify an intruder word, chosen based on difference in word embedding dimensions (Murphy et al., 2012; Fyshe et al., 2015; Faruqui et al., 2015).”
Visualizing and Understanding Neural Models in NLP (2015, background): “…neural models have been much more sig-niﬁcantly explored in vision, especially for Convolutional Neural Networks (CNNs or ConvNets) (Krizhevsky et al., 2012), multi-layer neural networks in which the original matrix of image pixels is convolved and pooled as it is passed on to hidden…”
Understanding Neural Networks through Representation Erasure (2016, background): “…2013; Karpathy et al., 2015), interpreting semantic dimensions by presenting humans with a list of words and asking them to choose outliers (Fyshe et al., 2015; Murphy et al., 2012), linking dimensions with semantic lexicons or word properties (Faruqui et al., 2014; Herbelot and Vecchi,…”
Is Attention Interpretable? (2019, background): “Yet another line of work focuses on aligning models with human feedback for what is interpretable (Fyshe et al., 2015; Subramanian et al., 2017), which could reﬁne our idea of what deﬁnes a high-quality explanation derived from attention.”
Sparse Overcomplete Word Vector Representations (2015, background): “Nonnegativity in the feature space has often been shown to correspond to interpretability (Lee and Seung, 1999; Cichocki et al., 2009; Murphy et al., 2012; Fyshe et al., 2014; Fyshe et al., 2015).”
Feuding Families and Former Friends: Unsupervised Learning for Dynamic Fictional Relationships (2016, background): “Other applications for which researchers have prioritized interpretable vector representations include text-to-vision mappings (Lazari-dou et al., 2014) and word embeddings (Fyshe et al., 2015; Faruqui et al., 2015).”
Word meaning in minds and machines (2020, background): “These attempts have had some successes in accounting for various human judgments (Fyshe et al., 2015; Günther & Marelli, 2016; Marelli et al., 2017; Vecchi et al., 2016); for example, one may assess a model by computing the similarity of its interpretations of two phrases and seeing if they…”
Measuring Compositionality in Representation Learning (2019, background): “A long line of work in natural language processing (Coecke et al., 2010; Baroni & Zamparelli, 2010; Clark, 2012; Fyshe et al., 2015) focuses on learning composition functions to produce distributed representations of phrases and sentences—that is, for purposes of modeling rather than evaluation.”
Learning Monolingual Compositional Representations via Bilingual Supervision (2016, methodology): “…prior work has focused either on composing words into short phrases (Mitchell and Lapata, 2010; Baroni and Zamparelli, 2010; Hermann et al., 2012; Fyshe et al., 2015), or on supervised task-speciﬁc composition functions (Socher et al., 2013; Iyyer et al., 2015; Rockt¨aschel et al., 2016; Iyyer…”
On internal language representations in deep learning: an analysis of machine translation and speech recognition (2018, background): “Using human behavioral experiments may be a reasonable proxy [60, 113, 249].”