No Training Required: Exploring Random Encoders for Sentence Classification

By John Wieting, Douwe Kiela. 2019

105 citations recorded by the source corpus.

We explore various methods for computing sentence representations from pre-trained word embeddings without any training, i.e., using nothing but random parameterizations. Our aim is to put sentence embeddings on more solid footing by 1) looking at how much modern sentence embeddings gain over random methods---as it turns out, surprisingly little; and by 2) providing the field with more appropriate baselines going forward---which are, as it turns out, quite strong. We also make important observations about proper experimental protocol for sentence classification evaluation, together with recommendations for future research.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Adversarial NLI: A New Benchmark for Natural Language Understanding (2019, background): “Many tasks do not actually require sophisticated linguistic reasoning, as shown by the surprisingly good performance of random encoders (Wieting and Kiela, 2019).”
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (2018, background): “Random Encoder A number of recent works have noted that randomly initialized, untrained LSTMs can obtain surprisingly strong downstream task performance (Zhang and Bowman, 2018; Wieting and Kiela, 2019; Tenney et al., 2019b).”
Pre-training via Paraphrasing (2020, methodology): “Using the same encoder for both the target and evidence documents allows even random models to compute meaningful similarity functions, as documents with high lexical overlap are more likely to be projected to more similar representations [Wieting and Kiela, 2019].”
Information-Theoretic Probing with Minimum Description Length (2020, methodology): “Probes using these representations show surprisingly strong performance for both token (Zhang and Bowman, 2018) and sentence (Wieting and Kiela, 2019) representations.”
Do Attention Heads in BERT Track Syntactic Dependencies? (2019, background): “has shown that randomly initialized sentence encoders perform surprisingly well on a suite of NLP tasks (Zhang and Bowman, 2018; Wieting and Kiela, 2019).”
Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text (2020, background): “We note that random sentence vectors are a competitive baseline on many language classiﬁcation tasks [Wieting and Kiela, 2019].”
Similarity Analysis of Contextual Word Representation Models (2020, background): “Still, it may explain why random models some-times generate useful features (Wieting and Kiela, 2019).”
What do you mean, BERT? Assessing BERT as a Distributional Semantics Model (2019, methodology): “Yet this methodology has been criticized as potentially conﬂicting with the intended purpose of studying the representations themselves (Wieting and Kiela, 2019; Cover, 1965); cf. also Hewitt and Liang (2019) for a discussion.”
Language Models are Unsupervised Multitask Learners (2019, background): “Much research has been dedicated to learning (Hill et al., 2016), understanding (Levy & Goldberg, 2014), and critically evaluating (Wieting & Kiela, 2019) the representations of both supervised and unsupervised pre-training methods.”
Sentence Meta-Embeddings for Unsupervised Semantic Textual Similarity (2019, background): “Wieting and Kiela (2019) warn that high-dimensional sentence embeddings can have an advantage over low-dimensional ones.”
Deep Contextualized Self-training for Low Resource Dependency Parsing (2019, background): “Indeed, in recent papers untrained LSTMs with random weights substantially enhanced model performance (Wang et al., 2019; Zhang and Bowman, 2018; Tenney et al., 2019; Wieting and Kiela, 2019).”