Evaluating Neural Word Representations in Tensor-Based Compositional Settings

By Dmitrijs Milajevs, Dimitri Kartsaklis, Mehrnoosh Sadrzadeh, Matthew Purver. 2014

116 citations recorded by the source corpus.

We provide a comparative study between neural word representations and traditional vector spaces based on cooccurrence counts, in a number of compositional tasks. We use three different semantic spaces and implement seven tensor-based compositional models, which we then test (together with simpler additive and multiplicative approaches) in tasks involving verb disambiguation and sentence similarity. To check their scalability, we additionally evaluate the spaces using simple compositional methods on larger-scale tasks with less constrained language: paraphrase detection and dialogue act tagging. In the more constrained tasks, co-occurrence vectors are competitive, although choice of compositional method is important; on the largerscale tasks, they are outperformed by neural word embeddings, which show robust, stable performance across the tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Minlie Huang
3. Mohit Bansal
4. Kyunghyun Cho
5. Li Deng
6. Zhiyuan Liu
7. Xiaodong He
8. Yejin Choi

In Computer Vision and Pattern Recognition

1. Jianfeng Gao
2. Luc Van Gool
3. Dacheng Tao
4. Trevor Darrell
5. Xiaogang Wang
6. Chunhua Shen
7. Andrew Zisserman
8. Ming-Hsuan Yang

In Natural Language Processing

1. Jianfeng Gao
2. Xiaodong Liu
3. Minlie Huang
4. Xiaodong He
5. Mohit Bansal
6. Kyunghyun Cho
7. Ivan Vulic
8. Yejin Choi

In Evaluation & Benchmarks

1. Ivan Vulic
2. Kyunghyun Cho
3. Jianfeng Gao
4. Anna Korhonen
5. Xiaodong Liu
6. Felix Hill
7. Jun Zhao
8. Xiaodong He

Citations in the corpus

• Learning Distributed Representations of Sentences from Unlabelled Data (2016, background): “See the contrasting conclusions in (Mitchell and Lapata, 2008; Clark and Pulman, 2007; Baroni et al., 2014a; Milajevs et al., 2014) among others.”
• Representation Learning Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval (2015, background): “Recently, learningbased approaches inspired by neural networks, especially DNNs, have gained in prominence, due to their favorable performance (Huang et al., 2013; Baroni et al., 2014; Milajevs et al., 2014).”
• Word Embedding based Edit Distance (2018, methodology): “Finally, cosine between the vectors of two strings is calculated and utilized (Mitchell and Lapata, 2008, Milajevs et al., 2014).”
• How to Generate a Good Word Embedding (2015, background): “However, Milajevs et al. [21] proposed that the co-occurrence matrix approaches should be more efficient than predict models for semantic compositional tasks.”
• Learning to Understand Phrases by Embedding the Dictionary (2015, unknown): “Both methods of composition were suggested in a recent study on building phrase representations from word embeddings (Milajevs et al., 2014).”
• Monolingual and Cross-Lingual Information Retrieval Models Based on (Bilingual) Word Embeddings (2015, background): “…in the embedding spaces (e.g., [26, 20]) 4 , we opt for simple addition (denoted by +) from [29] as the compositional operator, due to its simplicity, the ease of applicability considering the bag-of-words assumption, as well as its solid performance in various compositional tasks [29, 27].”