Directional Skip-Gram: Explicitly Distinguishing Left and Right Context for Word Embeddings

By Yan Song, Shuming Shi, Jing Li, Haisong Zhang. North American Chapter of the Association for Computational Linguistics, 2018

343 citations recorded by the source corpus.

In this paper, we present directional skip-gram (DSG), a simple but effective enhancement of the skip-gram model by explicitly distinguishing left and right context in word prediction. In doing so, a direction vector is introduced for each word, whose embedding is thus learned by not only word co-occurrence patterns in its context, but also the directions of its contextual words. Theoretical and empirical studies on complexity illustrate that our model can be trained as efficient as the original skip-gram model, when compared to other extensions of the skip-gram model. Experimental results show that our model outperforms others on different datasets in semantic (word similarity measurement) and syntactic (part-of-speech tagging) evaluations, respectively.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/ab096cffb36bb74026c3de0f2e7de40bf4b788ad

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Rui Liu
2. Berrak Sisman
3. Haizhou Li
4. Guanglai Gao
5. Chujie Zheng
6. Minlie Huang
7. Zexin Cai
8. Yuyang Nie

Citations in the corpus

• Modeling Prosodic Phrasing With Multi-Task Learning in Tacotron-Based TTS (2020, methodology): “For Chinese, we use the Tencent AI Lab embedding database for Chinese Words and Phrases [41].”
• Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning (2021, methodology): “We choose an Embedding Corpus[28] that covers over 8 million Chinese words and phrases, to assist us reveal the relevance between words.”
• Cross-modal Memory Networks for Radiology Report Generation (2022, methodology): “Since the quality of input representation plays an important role in model performance (Pennington et al., 2014; Song et al., 2017, 2018; Peters et al., 2018; Song and Shi, 2018; Devlin et al., 2019; Song et al., 2021), the encoder-decoder in our model is built upon standard Transformer (which is a…”
• Named Entity Recognition for Social Media Texts with Semantic Augmentation (2020, methodology): “, 2019); for Chinese, we use Tencent Embedding (Song et al., 2018b), and ZEN (Diao et al.”
• ChID: A Large-scale Chinese IDiom Dataset for Cloze Test (2019, methodology): “We set the vocabulary size to 100K and used the 200-dimensional word embeddings initialized by Song et al. (2018).”
• Polyphone Disambiguation for Mandarin Chinese Using Conditional Neural Network with Multi-level Embedding Features (2019, methodology): “This pre-trained model was trained with large-scale text collected from news, webpages, and novels using directional skip-gram [29, 30].”