Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
By Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, Christopher Potts. 2013
9,576 citations recorded by the source corpus.
Semantic word spaces have been very useful but cannot express the meaning of longer phrases in a principled way. Further progress towards understanding compositionality in tasks such as sentiment detection requires richer supervised training and evaluation resources and more powerful models of composition. To remedy this, we introduce a Sentiment Treebank. It includes fine grained sentiment labels for 215,154 phrases in the parse trees of 11,855 sentences and presents new challenges for sentiment compositionality. To address them, we introduce the Recursive Neural Tensor Network. When trained on the new treebank, this model outperforms all previous methods on several metrics. It pushes the state of the art in single sentence positive/negative classification from 80% up to 85.4%. The accuracy of predicting fine-grained sentiment labels for all phrases reaches 80.7%, an improvement of 9.7% over bag of features baselines. Lastly, it is the only model that can accurately capture the effects of negation and its scope at various tree levels for both positive and negative phrases.
Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Yoshua Bengio
- Yejin Choi
- Graham Neubig
- Noah A. Smith
In Natural Language Processing
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Graham Neubig
- Yejin Choi
- Noah A. Smith
- Percy Liang
- Kyunghyun Cho
In Representation & Self-Supervised Learning
- Yoshua Bengio
- Sebastian Ruder
- Jianfeng Gao
- Ivan Vulic
- Ruslan Salakhutdinov
- Percy Liang
- Graham Neubig
- Quoc V. Le
Citations in the corpus
- A Structured Self-attentive Sentence Embedding (2017, methodology): “This includes SkipThought vectors (Kiros et al., 2015), ParagraphVector (Le & Mikolov, 2014), recursive auto-encoders (Socher et al., 2011; 2013), Sequential Denoising Autoencoders (SDAE), FastSent (Hill et al., 2016), etc.”
- Long Short-Term Memory-Networks for Machine Reading (2016, methodology): “We used the Stanford Sentiment Treebank (Socher et al., 2013), which contains fine-grained sentiment labels (very positive, positive, neutral, negative, very negative) for 11,855 sentences.”
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2018, methodology): “Similarly, with the exception of InferSent, the sentence representation models are outperformed on SST by our BiLSTM and its non-CoVe variants.”
- Deep Contextualized Word Representations (2018, methodology): “Sentiment analysis The fine-grained sentiment classification task in the Stanford Sentiment Treebank (SST-5; Socher et al., 2013) involves selecting one of five labels (from very negative to very positive) to describe a sentence from a movie review.”
- Learned in Translation: Contextualized Word Vectors (2017, methodology): “Recent work in NLP has continued in this direction by using pretrained word representations to improve models for entailment [Bowman et al., 2014], sentiment analysis [Socher et al., 2013], summarization [Nallapati et al., 2016], and question answering [Seo et al., 2017, Xiong et al., 2017].”
- Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning (2017, background): “These encoders are typically implemented as convolutional (Kim, 2014), recursive (Socher et al., 2013), or recurrent neural networks (Mikolov et al., 2010) operating over a sentence’s words or characters (Zhang et al., 2015; Kim et al., 2016).”
- Semi-supervised Sequence Learning (2015, methodology): “We compare our method to their reported results [29] on sente nce-level classification.”
- A large annotated corpus for learning natural language inference (2015, background): “…at capturing relations based in similarity, and have proven effective at modeling simple dimensions of meaning like evaluative sentiment (e.g., Socher et al. 2013), but it is less clear that they can be trained to support the full range of logical and commonsense inferences required for NLI…”
- Skip-Thought Vectors (2015, background): “In recent years, several approaches have been developed for learning composition operators that map word vectors to sentence vectors including recursive networks [1], recurrent networks [2], convolutional networks [3, 4] and recursive-convolutional methods [5, 6] among others.”
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, background): “…meant to test general language understanding abilities: • Sentence acceptability judgment (CoLA [Warstadt et al., 2018]) • Sentiment analysis (SST-2 [Socher et al., 2013]) 6https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words 7https://pypi.org/project/langdetect/…”
- RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, background): “, 2018), Stanford Sentiment Treebank (SST) (Socher et al., 2013), Microsoft Research Paragraph Corpus (MRPC) (Dolan and Brockett, 2005), Semantic Textual Similarity Benchmark (STS) (Agirre et al.”
- Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “To do so, we used the sentences from the Stanford Sentiment Treebank dataset (Socher et al., 2013) and rendered them into images, with black texts on a white background, in a 448×448 resolution.”
- Adversarial Training for Large Neural Language Models (2020, methodology): “, 2018) and SST (Socher et al., 2013) from GLUE (Wang et al.”
- Pretrained Transformers Improve Out-of-Distribution Robustness (2020, methodology): “We utilize four sentiment analysis datasets: • We use SST-2, which contains pithy expert movie reviews (Socher et al., 2013), and IMDb (Maas et al.”
- Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, background): “We then finetune each model using the R O BERT A finetuning hyperparameters (Liu et al., 2019b) on MNLI and SST-2.”