Distributional vectors encode referential attributes

By Abhijeet Gupta, Gemma Boleda, Marco Baroni, Sebastian Padó. 2015

98 citations recorded by the source corpus.

Distributional methods have proven to excel at capturing fuzzy, graded aspects of meaning (Italy is more similar to Spain than to Germany). In contrast, it is difficult to extract the values of more specific attributes of word referents from distributional representations, attributes of the kind typically found in structured knowledge bases (Italy has 60 million inhabitants). In this paper, we pursue the hypothesis that distributional vectors also implicitly encode referential attributes. We show that a standard supervised regression model is in fact sufficient to retrieve such attributes to a reasonable degree of accuracy: When evaluated on the prediction of both categorical and numeric attributes of countries and cities, the model consistently reduces baseline error by 30%, and is not far from the upper bound. Further analysis suggests that our model is able to “objectify” distributional representations for entities, anchoring them more firmly in the external world in measurable ways.

Subfields: Cultural Studies, Statistical and Nonlinear Physics, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cultural Studies

In Statistical and Nonlinear Physics

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Probing Classifiers: Promises, Shortcomings, and Advances (2021, background): “Perhaps the ﬁrst studies that can be cast in the framework of probing classiﬁers are by Köhn (2015) and Gupta et al. (2015), who trained classiﬁers on static word embeddings to predict various morphological, syntactic, and semantic properties.”
Analysis Methods in Neural Language Processing: A Survey (2018, background): “(2018a) also verified that their examples do not contain annotation artifacts, a potential problem noted in recent studies (Gururangan et al., 2018; Poliak et al., 2018b). contrastive pairs evaluation of Sennrich (2017). Automatic evaluation metrics are cheap to obtain and can be calculated on a lar”
Probing for semantic evidence of composition by means of simple classification tasks (2016, methodology): “tive neuroscience studies of language understanding (Frankland and Greene, 2015), as well as work in NLP that has previously made use of classification accuracy for assessing information captured in vector representations (Gupta et al., 2015).”
Inducing Relational Knowledge from BERT (2019, background): “Similarly, Gupta et al. (2015) show that word vectors can to some extent predict ordinal attributes of cities and countries.”
Pre-trained models for natural language processing: A survey (2020, background): “d that distributional word representations are good at predicting taxonomic properties (e.g., dog is an animal) but fail to learn attributive properties (e.g., swan is white). Similarly, Gupta et al. [54] showed that word2vec embeddings implicitly encode referential attributes of entities. The distr”
Probing Classifiers: Promises, Shortcomings, and Alternatives (2021, background): “Perhaps the ﬁrst studies that can be cast in the framework of probing classiﬁers are by Köhn (2015) and Gupta et al. (2015), who trained classiﬁers on static word embeddings to predict various morphological, syntactic, and semantic properties.”
Formal Distributional Semantics: Introduction to the Special Issue (2016, background): “Recent work has started focusing on this question: Gupta et al. (2015) show that vectors of single entities (cities and countries) can be linked to referential attributes, as expressed by a formal ontology; Herbelot (2015) gives a model for constructing entity vectors from concept-level…”
On internal language representations in deep learning: an analysis of machine translation and speech recognition (2018, background): “eludes word embeddings [138, 188, 279], RNN hidden states or gate activations [102, 278, 306, 335, 345], and sentence embeddings [2, 3, 47, 81, 106, 117].”
Building a shared world: mapping distributional to model-theoretic semantic spaces (2015, methodology): “Note that we do not in principle reject the possibility to learn cardinalities from distributional data (for an example of this, see Gupta et al. (2015)).”
Language Models Represent Space and Time (2023, methodology): “We also perform a series of neuron ablation and intervention experiments in Appendix B to verify the importance of these neurons in spatial and temporal modeling. et al., 2015; Konkol et al., 2017).”