The Role of Context Types and Dimensionality in Learning Word Embeddings

By Oren Melamud, David McClosky, Siddharth Patwardhan, Mohit Bansal. 2016

126 citations recorded by the source corpus.

We provide the first extensive evaluation of how using different types of context to learn skip-gram word embeddings affects performance on a wide range of intrinsic and extrinsic NLP tasks. Our results suggest that while intrinsic tasks tend to exhibit a clear preference to particular types of contexts and higher dimensionality, more careful tuning is required for finding the optimal settings for most of the extrinsic tasks that we considered. Furthermore, for these extrinsic tasks, we find that once the benefit from increasing the embedding dimensionality is mostly exhausted, simple concatenation of word embeddings, learned with different context types, can yield further performance gains. As an additional contribution, we propose a new variant of the skip-gram model that learns word embeddings from weighted contexts of substitute words.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Ivan Vulic
2. Xipeng Qiu
3. Yonatan Belinkov
4. Xuanjing Huang
5. Douwe Kiela
6. Ido Dagan
7. Iryna Gurevych
8. Jianfeng Gao

In Natural Language Processing

1. Ivan Vulic
2. Douwe Kiela
3. Xipeng Qiu
4. Iryna Gurevych
5. Xuanjing Huang
6. Jacob Goldberger
7. Yonatan Belinkov
8. Ido Dagan

In Representation & Self-Supervised Learning

1. Ido Dagan
2. Jacob Goldberger
3. Oren Melamud
4. Ivan Vulic
5. Douwe Kiela
6. Peng Qian
7. Xipeng Qiu
8. Xuanjing Huang

Citations in the corpus

• context2vec: Learning Generic Context Embedding with Bidirectional LSTM (2016, methodology): “This was done in tasks, such as chunking, NER, semantic role labeling, and co-reference resolution (Turian et al., 2010; Collobert et al., 2011; Melamud et al., 2016), mostly by considering the embeddings of words in a window around the target of interest.”
• Investigating Language Universal and Specific Properties in Word Embeddings (2016, background): “Melamud et al. (2016) investigates the influence of context type and vector dimension on word embedding.”
• Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets (2019, background): “Different from word embeddings, it allows the meaning of a word to change according to the context in which it is used (Melamud et al., 2016; Peters et al., 2017; Devlin et al., 2019; Dai et al., 2019).”
• Evaluation by Association: A Systematic Study of Quantitative Word Association Evaluation (2017, background): “The construction of ground truth evaluation sets that reflect these relations, such as SimLex-999 (Hill et al., 2015), SimVerb-3500 (Gerz et al., 2016), MEN (Bruni et al., 2014) or Rare Words (Luong et al., 2013), relies on manually constructed guidelines that trigger subjective human interpretation ”