Semantic Specialisation of Distributional Word Vector Spaces using Monolingual and Cross-Lingual Constraints

By Nikola Mrksic, Ivan Vulic, Diarmuid Ó Séaghdha, Ira Leviant, Roi Reichart, Milica Gasic, Anna Korhonen, Susannah Young. 2017

219 citations recorded by the source corpus.

We present Attract-Repel, an algorithm for improving the semantic quality of word vectors by injecting constraints extracted from lexical resources. Attract-Repel facilitates the use of constraints from mono- and cross-lingual resources, yielding semantically specialised cross-lingual vector spaces. Our evaluation shows that the method can make use of existing cross-lingual lexicons to construct high-quality vector spaces for a plethora of different languages, facilitating semantic transfer from high- to lower-resource ones. The effectiveness of our approach is demonstrated with state-of-the-art results on semantic similarity datasets in six languages. We next show that Attract-Repel-specialised vectors boost performance in the downstream task of dialogue state tracking (DST) across multiple languages. Finally, we show that cross-lingual vector spaces produced by our algorithm facilitate the training of multilingual DST models, which brings further performance improvements.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling (2018, background): “The corpus was later extended to additional two languages for cross-lingual research (Mrkšić et al., 2017b).”
A Survey of Cross-lingual Word Embedding Models (2017, methodology): “Other striking findings include P@1 scores for Russian-Ukrainian (0.84 vs. 0.59), Czech-Slovak (0.82 vs. 0.59), Serbian-Croatian (0.78 vs. 0.56), or Danish-Norwegian (0.73 vs. 0.67).”
Informing Unsupervised Pretraining with External Linguistic Knowledge (2019, background): “It also further validates intuitions from relevant work on specialising static word embeddings (Wieting et al., 2015; Mrkˇsi´c et al., 2017) that steering distributional models towards capturing true semantic similarity (as also done here) has a positive impact on language understanding…”
Embedding Words and Senses Together via Joint Knowledge-Enhanced Training (2016, background): “ing low similarity scores for antonym pairs (Pham et al., 2015; Schwartz et al., 2015; Nguyen et al., 2016; Mrksic et al., 2017), but this is outside the scope of this paper.”
From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers (2020, background): “While reliable MT hinges on availability of large parallel corpora, transfer via multilingual KBs (Camacho-Collados et al., 2016; Mrkšić et al., 2017) is impaired by the limited KB coverage and inaccurate entity linking (Moro et al., 2014; Raiman and Raiman, 2018).”
A General Framework for Implicit and Explicit Debiasing of Distributional Word Vector Spaces (2019, methodology): “It is inspired by the work on semantic specialization of word embeddings (Mrkšić et al. 2017; Glavaš and Vulić 2018): but instead of using linguistic constraints (e.g., synonyms), we “specialize” the vector space by leveraging debiasing constraints.”
From Word to Sense Embeddings: A Survey on Vector Representations of Meaning (2018, background): “In contrast, knowledge-based representations are already linked to entries in a sense inventory, which enables a higher interpretability, as these entries are generally associated with deﬁnitions, examples, images and often relations with other concepts (e.g., WordNet) and translations (e.g., BabelN”
Lightly-supervised Representation Learning with Global Interpretability (2018, methodology): “A similar approach is proposed by (Mrkšić et al., 2017), who use an objective function modified with Attract and Repel components to fine-tune word embeddings with synonym and antonym pairs.”
Neural Sequence Learning Models for Word Sense Disambiguation (2017, methodology): “…WSD (Section 6.2), we experimented, instead, with two different configurations of the embedding layer: the pre-trained bilingual embeddings by Mrkšić et al. (2017) for all the language pairs of interest (EN-IT, EN-FR, EN-DE, and EN-ES), and the pre-trained multilingual 512-dimensional…”
Learning Knowledge Bases with Parameters for Task-Oriented Dialogue Systems (2020, background): “…the resource scarcity challenges in target domains (Bapna et al., 2017; Shah et al., 2019; Wu et al., 2019a; Liu et al., 2020) and target languages (Mrkˇsi´c et al., 2017; Schuster et al., 2019; Chen et al., 2018; Liu et al., 2019b), and large pre-trained language models are shown to possess the…”
Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representations (2018, methodology): “Other approaches predict mono-and bilingual context using word alignment information as an extension to the standard skip-gram model (Luong et al., 2015) or inject cross-lingual synonymy and antonymy constraints similar as in the monolingual setting (Mrkˇsi´c et al., 2017).”
Neural Natural Language Inference Models Enhanced with External Knowledge (2017, background): “…in neural networks for other NLP tasks, including word embedding (Chen et al., 2015; Faruqui et al., 2015; Liu et al., 2015; Wieting et al., 2015; Mrksic et al., 2017), machine translation (Shi et al., 2016; Zhang et al., 2017b), language modeling (Ahn et al., 2016), and dialogue systems (Chen…”
ConveRT: Efficient and Accurate Conversational Representations from Transformers (2019, background): “They mitigate the requirements for explicit task-specific semantics hand-crafted by domain experts (Henderson et al., 2014; Mrkšić et al., 2015, 2017). large collection of responses in a joint semantic space, and then retrieve the most relevant response by matching the query representation…”
Semantically Conditioned Dialog Response Generation via Hierarchical Disentangled Self-Attention (2019, background): “(ii) belief state tracker (Williams et al., 2013; Mrksic et al., 2017a,b; Zhong et al., 2018; Chen et al., 2018), which is used to track user’s query constraint and formulate DB query to retrieve entries from a large database.”