A Multiplicative Model for Learning Distributed Text-Based Attribute Representations

By Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov. 2014

67 citations recorded by the source corpus.

In this paper we propose a general framework for learning distributed representations of attributes: characteristics of text whose representations can be jointly learned with word embeddings. Attributes can correspond to a wide variety of concepts, such as document indicators (to learn sentence vectors), language indicators (to learn distributed language representations), meta-data and side information (such as the age, gender and industry of a blogger) or representations of authors. We describe a third-order model where word context and attribute vectors interact multiplicatively to predict the next word in a sequence. This leads to the notion of conditional word similarity: how meanings of words change when conditioned on different attributes. We perform several experimental tasks including sentiment classification, cross-lingual document classification, and blog authorship attribution. We also qualitatively evaluate conditional word neighbours and attribute-conditioned text generation.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models (2014, methodology): “In [30], the conditioning vector u is referred to as an attribute and using a third-order model of words allows one to model conditional similarity: how meanings of words change as a function of the attributes they’re conditioned on.”
Contextual Sequence Modeling for Recommendation with Recurrent Neural Networks (2017, background): “Multiplicative interaction offers tighter binding of context into item representation and is similar to [11].”
Word Representations via Gaussian Embedding (2014, background): “Other work in multiplicative tensor factorization for word embeddings (Kiros et al., 2014) and metric learning (Xing et al., 2002) learns some combinations of representations, clusters, and a distance metric jointly; however, it does not effectively learn a distance function per item.”
Semantic Compositional Networks for Visual Captioning (2016, neutral): “Similar ideas have been exploited in [25, 30, 38, 39, 41, 50, 14].”
Multi-layer Representation Learning for Medical Concepts (2016, background): “been shown to improve the performance of various complex tasks across domains spanning from image processing [22, 17, 36], language modeling [3, 25], word embedding [26, 30], music information retrieval [31], sentiment analysis [32], and multi-modal learning of images and text [18].”
Estimating Embedding Vectors for Queries (2016, methodology): “AWE, averaging or summing the embedding vectors of all query terms, has been previously used to construct embedding vectors of queries [21, 24, 28, 35, 39].”
User Modeling with Neural Network for Review Rating Prediction (2015, background): “Continuous user representation is also exploited in [Kiros et al., 2014; Perozzi et al., 2014] .”
Under review as a conference paper at ICLR 2016 (2015, background): “Other notable extensions include Kiros et al. (2014), which employed tensors to incorporate meta-data into the embedding, and Pennington et al. (2014), which leveraged global context as well as local during learning.”
Multi-Task Cross-Lingual Sequence Tagging from Scratch (2016, methodology): “Cross-lingual joint training typically uses word alignments or parallel corpora to improve the performance on different languages (Kiros et al., 2014; Gouws et al., 2014).”
E-commerce in Your Inbox: Product Recommendations at Scale (2015, background): “More recently, the concept of distributed representations has been extended beyond word representations to sentences and paragraphs [10, 21], relational entities [6, 27], general text-based attributes [19], descriptive text of images [18], nodes in graph structure [25], and other applications.”
Latent Cross: Making Use of Context in Recurrent Recommender Systems (2018, methodology): “Note, we can also apply this operation after the RNN: The technique offered in [27] can be viewed as a special case where multiplicative relations are included at the very top of the network along with the softmax function to improve NLP tasks.”