Deep Unordered Composition Rivals Syntactic Methods for Text Classification

By Mohit Iyyer, Varun Manjunatha, Jordan L. Boyd-Graber, Hal Daumé Iii. 2015

842 citations recorded by the source corpus.

Many existing deep learning models for natural language processing tasks focus on learning the compositionality of their inputs, which requires many expensive computations. We present a simple deep neural network that competes with and, in some cases, outperforms such models on sentiment analysis and factoid question answering tasks while taking only a fraction of the training time. While our model is syntactically-ignorant, we show significant improvements over previous bag-of-words models by deepening our network and applying a novel variant of dropout. Moreover, our model performs better than syntactic models on datasets with high syntactic variance. We show that our model makes similar errors to syntactically-aware models, indicating that for the tasks we consider, nonlinearly transforming the input is more important than tailoring a network to incorporate word order and syntax.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation (2017, background): “ECNU’s deep learning models are differentiated by their approach to sentence embeddings using either: averaged word embeddings, projected word embeddings, a deep averaging network (DAN) (Iyyer et al., 2015) or LSTM (Hochreiter and Schmidhuber, 1997).”
Learning Distributed Representations of Sentences from Unlabelled Data (2016, methodology): “The ‘word dropout’ effect when p o ≥ 0 has also been used as a regulariser for deep nets in supervised language tasks (Iyyer et al., 2015), and for large p x the objective is similar to word-level ‘debag-ging’ (Sutskever et al., 2011).”
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (2018, methodology): “This baseline is especially strong because our ELMo-style models use a skip connection from the input of the encoder to the output, allowing the task-specific component to see the input representations, yielding a model similar to Iyyer et al. (2015).”
Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates (2018, methodology): “Word dropout (Iyyer et al., 2015) is a simple approach for a bag-of-words representation, in which the embedding of a certain word sequence is simply calculated by averaging the word embeddings.”
Pretrained Transformers for Text Ranking: BERT and Beyond (2020, methodology): “As an alternative, Iyyer et al. [2015] proposed Deep Averaging Networks, which disregarded hierarchical structure to compute both sentence- as well as document-level representations based on averaging the embeddings of the individual words and then passing the results through feedforward layers.”
BPE-Dropout: Simple and Effective Subword Regularization (2019, methodology): “Data augmentation techniques in natural language processing include dropping to-kens at random positions or swapping tokens at close positions (Iyyer et al., 2015; Artetxe et al., 2018; Lample et al., 2018), replacing tokens at random positions with a placeholder token (Xie et al., 2017), replacing…”
On Measuring Social Biases in Sentence Encoders (2019, background): “Universal Sentence Encoder (USE): A variant of the deep averaging network (Iyyer et al., 2015), which passes an average of unigram and bi-gram embeddings in the sentence to a feedforward neural network to produce a 512-dimensional sentence encoding.”
Probing for semantic evidence of composition by means of simple classification tasks (2016, methodology): “Similarly, the concept of scope is critical to many key linguistic phenomena, including negation—the importance of which is widely acknowledged in NLP, in particular for applications such as sentiment analysis (Blunsom et al., 2013; Iyyer et al., 2015).”
Rationalizing Neural Predictions (2016, background): “, 2014), deep averaging network (Iyyer et al., 2015; Joulin et al., 2016) or a boosting classifier as the encoder.”
On Calibration of Modern Neural Networks (2017, methodology): “On 20 News and Reuters, we train Deep Averaging Networks (DANs) (Iyyer et al., 2015) with 3 feed-forward layers and Batch Normalization.”
Towards Universal Paraphrastic Sentence Embeddings (2015, methodology): “Our third model is the deep averaging network (DAN) of Iyyer et al. (2015).”
Universal Sentence Encoder (2018, methodology): “One makes use of the transformer (Vaswani et al., 2017) architecture, while the other is formulated as a deep averaging network (DAN) (Iyyer et al., 2015).”
Conversational AI: The Science Behind the Alexa Prize (2018, background): “Few teams such as Magnus (Prabhumoye et al., 2017) even built Finite State Machines (FSMs) (Wright, 2005) for addressing specific modules such as Movies, Sports, etc.”
Learning Semantic Textual Similarity from Conversations (2018, methodology): “For DAN, we employ a 3-layer DNN with layers of 300, 300, and 500 hidden units.”
A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence Classification (2015, background): “Convolutional Neural Networks (CNNs) have recently been shown to achieve impressive results on the practically important task of sentence categorization (Kim, 2014; Kalchbrenner et al., 2014; Wang et al., 2015; Goldberg, 2015; Iyyer et al., 2015; Zhang et al., 2016, 2017).”
Data Noising as Smoothing in Neural Network Language Models (2017, methodology): “The technique of randomly zero-masking input word embeddings for learning sentence representations has been proposed by Iyyer et al. (2015), Kumar et al. (2015), and Dai & Le (2015), and adopted by others such as Bowman et al. (2015).”
Deriving Neural Architectures from Sequence and Graph Kernels (2017, positive): “Following the recent work of DAN (Iyyer et al., 2015) and RLSTM (Tai et al., 2015), we use the publicly available 300-dimensional GloVe word vectors (Pennington et al., 2014).”
Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms (2018, methodology): “Moreover, several recent studies have shown empirically that the advantages of distinct compositional functions are highly dependent on the spe-ciﬁc task (Mitchell and Lapata, 2010; Iyyer et al., 2015; Zhang et al., 2015a; Wieting et al., 2015; Arora et al., 2016).”