Investigating Language Universal and Specific Properties in Word Embeddings

By Peng Qian, Xipeng Qiu, Xuanjing Huang. 2016

60 citations recorded by the source corpus.

Recently, many NLP tasks have benefited from distributed word representation. However, it remains unknown whether embedding models are really immune to the typological diversity of languages, despite the language-independent architecture. Here we investigate three representative models on a large set of language samples by mapping dense embedding to sparse linguistic property space. Experiment results reveal the language universal and specific properties encoded in various word representation. Additionally, strong evidence supports the utility of word form, especially for inflectional languages.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Computational Linguistics & Semantics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Computational Linguistics & Semantics

Citations in the corpus

Analysis Methods in Neural Language Processing: A Survey (2018, background): “A few studies compared different classifiers and found that deeper classifiers lead to overall better results, but do not alter the respective trends when comparing different models or components (Qian et al., 2016b; Belinkov, 2018).”
What do Neural Machine Translation Models Learn about Morphology? (2017, methodology): “Different units have been used, from word embeddings (Köhn, 2015; Qian et al., 2016b), through LSTM gates or states (Qian et al., 2016a), to sentence embeddings (Adi et al., 2016).”
Designing and Interpreting Probes with Control Tasks (2019, background): “As a counterpoint though, “complex” MLP probes have also been suggested since useful properties might be encoded non-linearly (Conneau et al., 2018), and they tend to report similar trends to simpler probes anyway (Belinkov et al., 2017; Qian et al., 2016).”
Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Tasks (2017, methodology): “We use a non-linear classifier because previous work found that it outperforms a linear classifier, while showing very similar trends (Qian et al., 2016b; Belinkov et al., 2017).”
Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systems (2017, background): “like vision and natural language processing, including word and sentence representations [22, 23, 24], machine translation [25, 26], and joint vision-language models [27].”
On internal language representations in deep learning: an analysis of machine translation and speech recognition (2018, background): “eludes word embeddings [138, 188, 279], RNN hidden states or gate activations [102, 278, 306, 335, 345], and sentence embeddings [2, 3, 47, 81, 106, 117].”
Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance? (2020, background): “…and the settings that are appropriate for probing (Alain and Bengio, 2016; Hewitt and Liang, 2019; Liu et al., 2019a; Conneau et al., 2018; Belinkov et al., 2017b; Qian et al., 2016; Voita and Titov, 2020), it is far from the only concern when interpreting the results of a probing experiment.”
What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models (2018, methodology): “This includes analyzing word and sentence embeddings (Adi et al. 2016; Qian, Qiu, and Huang 2016b; Conneau et al. 2018), recurrent neural network (RNN) states (Shi, Padhi, and Knight 2016; Wang, Chung, and Lee 2017), and NMT representations (Belinkov et al. 2017a; Belinkov et al. 2017b; Dalvi et…”
One Size Does Not Fit All: Comparing NMT Representations of Different Granularities (2019, methodology): “…analyzing word and sentence embeddings (Adi et al., 2017; Qian et al., 2016b; Ganesh et al., 2017; Conneau et al., 2018, among others), RNN states (Qian et al., 2016a; Shi et al., 2016; Wu and King, 2016; Wang et al., 2017), and NMT representations (Shi et al., 2016; Belinkov et al., 2017a), as…”
Analyzing Linguistic Knowledge in Sequential Model of Sentence (2016, methodology): “The basic method is to decode interpretable linguistic features from target neuron clusters, which has been used in (Köhn, 2015; Qian et al., 2016).”