Rotated Word Vector Representations and their Interpretability

By Sungjoon Park, Jinyeong Bak, Alice H. Oh. 2017

62 citations recorded by the source corpus.

Vector representation of words improves performance in various NLP tasks, but the high dimensional word vectors are very difficult to interpret. We apply several rotation algorithms to the vector representation of words to improve the interpretability. Unlike previous approaches that induce sparsity, the rotated vectors are interpretable while preserving the expressive performance of the original vectors. Furthermore, any prebuilt word vector representation can be rotated for improved interpretability. We apply rotation to skipgrams and glove and compare the expressive power and interpretability with the original vectors and the sparse overcomplete vectors. The results show that the rotated vectors outperform the original and the sparse overcomplete vectors for interpretability and expressiveness tasks.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yonatan Belinkov
2. James R. Glass
3. Siva Reddy
4. A. P. Sarath Chandar
5. Andreas Madsen
6. Percy Liang
7. Aykut Koç
8. Tolga Cukur

In Natural Language Processing

1. Yonatan Belinkov
2. James R. Glass
3. Siva Reddy
4. A. P. Sarath Chandar
5. Andreas Madsen
6. Aykut Koç
7. Tolga Cukur
8. Ihsan Utlu

In Representation & Self-Supervised Learning

1. Yonatan Belinkov
2. James R. Glass
3. A. P. Sarath Chandar
4. Aykut Koç
5. Siva Reddy
6. Andreas Madsen
7. Tolga Cukur
8. Ihsan Utlu

Citations in the corpus

• Analysis Methods in Neural Language Processing: A Survey (2018, background): “Since the evaluation is costly for high-dimensional representations, alternative automatic metrics were considered (Park et al., 2017; Senel et al., 2018).”
• Semantic Structure and Interpretability of Word Embeddings (2017, methodology): “Although interpretability evaluations used in [27] and [28] are free of human effort, they do not necessarily reflect human interpretations since they are directly calculated from the embeddings.”
• Post-hoc Interpretability for Neural NLP: A Survey (2021, methodology): “Park et al. [84] validate this method to be human-grounded by using the word intrusion test.”