Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms

By Dinghan Shen, Guoyin Wang, Wenlin Wang, Martin Renqiang Min, Qinliang Su, Yizhe Zhang, Chunyuan Li, Ricardo Henao, Lawrence Carin. 2018

351 citations recorded by the source corpus.

Many deep learning architectures have been proposed to model the compositionality in text sequences, requiring substantial number of parameters and expensive computations. However, there has not been a rigorous evaluation regarding the added value of sophisticated compositional functions. In this paper, we conduct a point-by-point comparative study between Simple Word-Embedding-based Models (SWEMs), consisting of parameter-free pooling operations, relative to word-embedding-based RNN/CNN models. Surprisingly, SWEMs exhibit comparable or even superior performance in the majority of cases considered. Based upon this understanding, we propose two additional pooling strategies over learned word embeddings: (i) a max-pooling operation for improved interpretability; and (ii) a hierarchical pooling operation, which preserves spatial (n-gram) information within text sequences. We present experiments on 17 datasets encompassing three tasks: (i) (long) document classification; (ii) text sequence matching; and (iii) short text tasks, including classification and tagging.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization (2018, neutral): “(1)Note that encoders based on RNN or pure word embedding [18] are also possible, nevertheless we limit our choice to CNN in this paper.”
Joint Embedding of Words and Labels for Text Classification (2018, methodology): “For the computational complexity, our model is almost same order as the most simple SWEM model, and is smaller than the CNN or LSTM by a factor of mh/K or h/K .”
Estimating Training Data Influence by Tracking Gradient Descent (2020, methodology): “We train a Simple Word-Embedding Model (SWEM) (Shen et al., 2018) for 60 epochs and use the default parameters of sentencepeice library as tokenizer (Kudo & Richardson, 2018) and achieve 95.5% on both training and test.”
NASH: Toward End-to-End Neural Architecture for Generative Semantic Hashing (2018, background): “…significant research has been made to learn continuous deep representations for words or documents (Mikolov et al., 2013; Kiros et al., 2015; Shen et al., 2018), discrete neural representations have been mainly explored in learning word embeddings (Shu and Nakayama, 2017; Chen et al., 2017).”
Every Document Owns Its Structure: Inductive Text Classification via Graph Neural Networks (2020, methodology): “We consider three types of models as baselines: (i) traditional deep learning methods including TextCNN (Kim, 2014) and TextRNN (Liu et al., 2016), (ii) simple but efficient strategies upon word features including fastText (Joulin et al., 2017) and SWEM (Shen et al., 2018), and (iii) graph-based met”
Improved Semantic-Aware Network Embedding with Fine-Grained Word Alignment (2018, methodology): “We experiment with three variants for our WANE model: ( i ) WANE: where the word embeddings of each text sequence are simply average to obtain the sentence representations, similar to (Joulin et al., 2016; Shen et al., 2018c).”
Teaching Syntax by Adversarial Distraction (2018, methodology): “More recently, FEVER introduced a fact veri-ﬁcation task, where claims are to be veriﬁed using all of Wikipedia.”
Knowledge-Enriched Transformer for Emotion Detection in Textual Conversations (2019, background): “Discarding early contextual utterances may cause information loss, but this loss is negligible because they only contribute the least amount of information (Su et al., 2018).”
No Training Required: Exploring Random Encoders for Sentence Classification (2019, background): “Indeed, state-of-the-art word embeddings on their own perform quite well with simple pooling mechanisms, as reported by Wieting et al. (2015); Arora et al. (2017) and Shen et al. (2018).”
Contextual and Non-Contextual Word Embeddings: an in-depth Linguistic Investigation (2020, methodology): “Similarly, but focused on the resolution of speciﬁc downstream tasks, Shen et al. (2018) compared a Single Word Embedding-based model (SWEM-based) with existing recurrent and convolutional networks using a suite of 17 NLP datasets, demonstrating that simple pooling operations over SWEM-based…”
Why Mean Pooling Works: Quantifying Second-Order Collapse in Text Embeddings (2026, unknown): “This simple aggregation has offered empirical advantages across embedding methods, from classical static token embeddings (Wieting et al., 2016; Shen et al., 2018) to modern contextualized embeddings from Transformer (Vaswani et al., 2017) encoders (Reimers and Gurevych, 2019).”