AutoExtend: Extending Word Embeddings to Embeddings for Synsets and Lexemes

By Sascha Rothe, Hinrich Schütze. 2015

290 citations recorded by the source corpus.

We present \textit{AutoExtend}, a system to learn embeddings for synsets and lexemes. It is flexible in that it can take any word embeddings as input and does not need an additional training corpus. The synset/lexeme embeddings obtained live in the same vector space as the word embeddings. A sparse tensor formalization guarantees efficiency and parallelizability. We use WordNet as a lexical resource, but AutoExtend can be easily applied to other resources like Freebase. AutoExtend achieves state-of-the-art performance on word similarity and word sense disambiguation tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

context2vec: Learning Generic Context Embedding with Bidirectional LSTM (2016, neutral): “In the Senseval-3 supervised WSD task, the best result is due to Ando (2006) and the second-best to Rothe and Sch¨utze (2015). context2vec is almost on par with these results, which were achieved with dedicated feature engineering and supervised machine learning models.”
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations (2016, methodology): “Canonicalization, or WSD (Pal and Saha, 2015), has been used in numerous applications, including machine translation, information retrieval, and information extraction (Rothe and Schütze, 2015, Leacock et al., 1998).”
Leveraging Pre-trained Checkpoints for Sequence Generation Tasks (2019, background): “(Huang et al., 2012; Rothe and Schütze, 2015)) before, but only in 2018 did training of the contextualized embeddings using large deep neural networks and an unsupervised training scheme become popular.”
Ultradense Word Embeddings by Orthogonal Transformation (2016, background): “In a tensor framework, Rothe and Sch¨utze (2015) transform the word embeddings to sense (synset) embeddings.”
Unsupervised Visual Sense Disambiguation for Verbs using Multimodal Embeddings (2016, background): “Granularity issues with WordNet for text-based WSD are well documented (Navigli, 2009).”
Embeddings for Word Sense Disambiguation: An Evaluation Study (2016, methodology): “…(cf. Section 3.6) and introduces the combined embeddings as a new feature in addition to the standard WSD features in IMS; and (2) AutoExtend (Rothe and Schütze, 2015), which constructs a whole new set of features based on vectors made from words, senses and synsets of WordNet and…”
Towards a Seamless Integration of Word Senses into Downstream NLP Applications (2017, background): “individual meanings of words (Chen et al., 2014; Johansson and Pina, 2015; Jauhar et al., 2015; Iacobacci et al., 2015; Rothe and Schütze, 2015; Camacho-Collados et al., 2016b; Mancini et al., 2016; Pilehvar and Collier, 2016).”
A La Carte Embedding: Cheap but Effective Induction of Semantic Feature Vectors (2018, methodology): “Embeddings of synsets, or sets of cognitive synonyms, and related entities such as senses and lexemes have been widely studied, often due to the desire to account for polysemy (Rothe and Schütze, 2015; Iacobacci et al., 2015).”
Linked Disambiguated Distributional Semantic Networks (2016, methodology): “These include approaches that modify Word2Vec [15] to learn sense embeddings [5], methods to enrich WordNet with embeddings for synsets and lexemes [21], acquire continuous word representations by combining random walks over knowledge bases and neural language models [11], or produce joint lexical a”
From Word to Sense Embeddings: A Survey on Vector Representations of Meaning (2018, methodology): “…Navigli, & Collier, 2017), document similarity (Wu & Giles, 2015), and word sense induction (Pelevina et al., 2016; Panchenko, Ruppert, Faralli, Ponzetto, & Biemann, 2017b) and disambiguation (Chen et al., 2014; Rothe & Sch¨utze, 2015; Camacho-Collados et al., 2016; Peters et al., 2018).”
Problems With Evaluation of Word Embeddings Using Word Similarity Tasks (2016, background): “However, there has been progress on obtaining multiple vectors per word-type to account for different word-senses (Reisinger and Mooney, 2010; Huang et al., 2012; Neelakantan et al., 2014; Jauhar et al., 2015; Rothe and Schütze, 2015).”
What do you mean, BERT? Assessing BERT as a Distributional Semantics Model (2019, methodology): “DSM s have also been computed for linguistic items other than words, e.g., word senses—based both on meaning inventories (Rothe and Sch¨utze, 2015) and word sense induction techniques (Bartunov et al., 2015)—or meaning exemplars (Reisinger and Mooney, 2010; Erk and Pad´o, 2010; Reddy et al., 2011).”
SenseBERT: Driving Some Sense into BERT (2019, methodology): “WordNet) have been employed, relying on hypernymity relations (Rothe and Schütze, 2015) or the gloss for each sense (Chen et al.”
Semantic Specialisation of Distributional Word Vector Spaces using Monolingual and Cross-Lingual Constraints (2017, background): “3 Fine-Tuning Pre-trained Vectors: Rothe and Schütze (2015) ﬁne-tune word vector spaces to improve the representations of synsets/lexemes found in WordNet.”
Embedding Words and Senses Together via Joint Knowledge-Enhanced Training (2016, background): “In order to obtain a representation for each sense of a word, a number of approaches have leveraged lexical resources to learn sense embeddings as a result of post-processing conventional word embeddings (Chen et al., 2014; Johansson and Pina, 2015; Jauhar et al., 2015; Rothe and Schütze, 2015; Pile”
Visual and Semantic Knowledge Transfer for Large Scale Semi-Supervised Object Detection (2016, methodology): “We observed that AutoExtend has indeed helped form better semantic relations between the desired categories: seal is now clustered with other animal categories like whale and turtle , and the nearest neighbors for ruler are now rubber eraser , power drill and pencil box .”
De-Conflated Semantic Representations (2016, methodology): “For this task, we can only compare against the publicly-available sense representations of Iacobacci et al. (2015), Rothe and Sch¨utze (2015), Pilehvar and Navigli (2015) and Chen et al. (2014) which are linked to the WordNet sense inventory.”
Supersense Embeddings: A Unified Model for Supersense Interpretation, Prediction, and Utilization (2016, background): “We are also aware that some of the previously proposed approaches for building word sense embeddings (Rothe and Sch¨utze, 2015; Chen et al., 2014; Iacobacci et al., 2015) could be eventually extended to supersenses.”