Unsupervised Word Sense Disambiguation Rivaling Supervised Methods

By David Yarowsky. 1995

This paper presents an unsupervised learning algorithm for sense disambiguation that, when trained on unannotated English text, rivals the performance of supervised techniques that require time-consuming hand annotations. The algorithm is based on two powerful constraints---that words tend to have one sense per discourse and one sense per collocation---exploited in an iterative bootstrapping procedure. Tested accuracy exceeds 96%.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Semi-Supervised Sequence Modeling with Cross-View Training (2018, methodology): “Self-training has historically been effective for NLP (Yarowsky, 1995; McClosky et al., 2006), but is less commonly used with neural models.”
Toolformer: Language Models Can Teach Themselves to Use Tools (2023, neutral): “…self-training and bootstrapping techniques to improve models has been investigated in various contexts, ranging from word sense disambiguation (Yarowsky, 1995), relation extraction (Brin, 1999; Agichtein and Gravano, 2000), parsing (McClosky et al., 2006), sequence generation (He et al.,…”
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference (2020, methodology): “The idea behind iPET – training multiple generations of models on data labeled by previous generations – bears resemblance to self-training and bootstrapping approaches for word sense disambiguation (Yarowsky, 1995), relation extraction (Brin, 1999; Agichtein and Gravano, 2000; Batista et al.”
Augmented Language Models: a Survey (2023, methodology): “As an alternative to standard fine-tuning, several authors propose to use bootstrapping techniques (e.g. Yarowsky, 1995; Brin, 1999) to leverage some form of indirect supervision.”
Data Augmentation for BERT Fine-Tuning in Open-Domain Question Answering (2019, methodology): “The roots of the distant supervision techniques we use trace back to at least the 1990s (Yarowsky, 1995; Riloff, 1996), although the term had not yet been coined.”
It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (2020, neutral): “, 2015) and self-training (e.g., Scudder, 1965; Yarowsky, 1995; Brin, 1999; McClosky et al., 2006) to easily combine several reformulations.”
PEER: A Collaborative Language Model (2022, methodology): “This enables us to use self-training techniques (e.g., Yarowsky, 1995; Sennrich et al., 2016; He et al., 2020a; Schick and Schütze, 2021a) for training PEER with synthetic plans, edits, explanations and documents.”
Revisiting Self-Training for Neural Sequence Generation (2019, methodology): “S is usually selected based on some conﬁdence scores (e.g. log probability) (Yarowsky, 1995) but it is also possible for S to be the whole pseudo parallel data (Zhu & Goldberg, 2009).”
Self-training Improves Pre-training for Natural Language Understanding (2020, methodology): “Self-training is a semi-supervised method which uses a teacher model, trained using labeled data, to create synthetic labels for unlabeled examples (Scudder, 1965; Yarowsky, 1995).”
Self-Training and Pre-Training are Complementary for Speech Recognition (2020, neutral): “This includes classical self-training [9, 10, 11] which demonstrated strong results [12, 13, 2, 14, 15] by pseudolabeling unannotated audio data and then retraining the final system with the additional labeled data.”
Improved Noisy Student Training for Automatic Speech Recognition (2020, background): “Self-training, where a teacher model is used to generate labels for the unlabeled set the student can train on, is one of the earliest methods in semi-supervised learning (e.g., [9, 10, 11]).”
Self-Training for End-to-End Speech Recognition (2019, methodology): “We revisit self-training [2, 3] in the context of sequenceto-sequence models with attention.”
Temporal Ensembling for Semi-Supervised Learning (2016, background): “There is a large body of previous work on semi-supervised learning (Zhu, 2005).”
AutoML: A Survey of the State-of-the-Art (2019, methodology): “[67] summarize the self-labeling methods into the following categories: selftraining [68, 69], co-training [70, 71], and co-learning [72].”
Meta-Learning for Semi-Supervised Few-Shot Classification (2018, background): “As for the literature on semi-supervised learning, while it is quite vast (Zhu, 2005; Chapelle et al., 2010), the most relevant category to our work is related to self-training (Yarowsky, 1995; Rosenberg et al., 2005).”