Effective Self-Training for Parsing

By David McClosky, Eugene Charniak, Mark Johnson. 2006

701 citations recorded by the source corpus.

We present a simple, but surprisingly effective, method of self-training a two-phase parser-reranker system using readily available unlabeled data. We show that this type of bootstrapping is possible for parsing when the bootstrapped parses are processed by a discriminative reranker. Our improved model achieves an f-score of 92.1%, an absolute 1.1% improvement (12% error reduction) over the previous best result for Wall Street Journal parsing. Finally, we provide some analysis to better understand the phenomenon.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Attention is All you Need (2017, neutral): “During inference, we [40] semi-supervised 91.3 Huang & Harper (2009) [14] semi-supervised 91.3 McClosky et al. (2006) [26] semi-supervised 92.1 Vinyals & Kaiser el al.”
Grammar as a Foreign Language (2014, background): “ WSJ only N/A 90.4 Petrov et al. (2010) ensemble [14] WSJ only 92.5 91.8 Zhu et al. (2013) [13] semi-supervised N/A 91.3 Huang & Harper (2009) [15] semi-supervised N/A 91.3 McClosky et al. (2006) [16] semi-supervised 92.4 92.1 Huang & Harper (2010) ensemble [17] semi-supervised 92.8 92.4 Tab”
Semi-Supervised Sequence Modeling with Cross-View Training (2018, methodology): “We achieve excellent results across seven NLP tasks, especially when CVT is combined with multi-task learning.”
Toolformer: Language Models Can Teach Themselves to Use Tools (2023, background): “…various contexts, ranging from word sense disambiguation (Yarowsky, 1995), relation extraction (Brin, 1999; Agichtein and Gravano, 2000), parsing (McClosky et al., 2006), sequence generation (He et al., 2020), few-shot text classification (Schick and Schütze, 2021a) and retrieval (Izacard and…”
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference (2020, background): “, 2015), parsing (McClosky et al., 2006; Reichart and Rappoport, 2007; Huang and Harper, 2009), machine translation (Hoang et al.”
Improving Neural Machine Translation Models with Monolingual Data (2015, background): “Another similar avenue of research is self-training (McClosky et al., 2006; Schwenk, 2008).”
Break-It-Fix-It: Unsupervised Learning for Program Repair (2021, neutral): “Self-training (Lee, 2013; McClosky et al., 2006; Kumar et al., 2020; Xie et al., 2021) applies a trained model to unlabeled data, obtains predicted targets (pseudolabels), and uses them as extra training examples.”
It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (2020, neutral): “, 2015) and self-training (e.g., Scudder, 1965; Yarowsky, 1995; Brin, 1999; McClosky et al., 2006) to easily combine several reformulations.”
FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence (2020, methodology): “, using a model’s predictions to obtain artificial labels for unlabeled data) has led it to be applied in many domains including NLP [31], object detection [44], image classification [25, 55], domain”
Revisiting Self-Training for Neural Sequence Generation (2019, methodology): “In the ﬁeld of natural language processing, some early work have successfully applied self-training to word sense disambiguation (Yarowsky, 1995) and parsing (McClosky et al., 2006; Reichart & Rappoport, 2007; Huang & Harper, 2009).”
Self-training Improves Pre-training for Natural Language Understanding (2020, background): “…applied to a variety of tasks, including image recognition (Yalniz et al., 2019; Xie et al., 2020; Zoph et al., 2020), automatic speech recognition (Syn-naeve et al., 2019; Kahn et al., 2020; Park et al., 2020), sequence generation (He et al., 2019), and parsing (McClosky et al., 2006).”
Natural Language Processing (Almost) from Scratch (2011, background): “• Self-training approaches, such as (Ueffing et al., 2007) for machine translation, and (McClosky et al., 2006) for parsing.”
Self-Training for End-to-End Speech Recognition (2019, methodology): “We revisit self-training [2, 3] in the context of sequenceto-sequence models with attention.”
Speaker-Follower Models for Vision-and-Language Navigation (2018, methodology): “The semi-supervised approach we use is related to model bootstrapping techniques such as self-training [42, 31] and co-training [6] at a high-level.”
Structured Generative Models of Natural Source Code (2014, background): “PCFGs and their variants are components of state-of-the-art parsers of English (McClosky et al., 2006), and many variants have been explored.”
Multi-domain Dialog State Tracking using Recurrent Neural Networks (2015, methodology): “Researchers have investigated methods for mitigating this problem, with NLP applications in parsing (McClosky et al., 2006; McClosky et al., 2010), sentiment analysis (Blitzer et al., 2007; Glorot et al., 2011) and many other tasks.”
Unbounded Dependency Recovery for Parser Evaluation (2009, methodology): “McClosky et al. (2006) report an F-score of 92.1% using self-training applied to the reranker of Charniak and Johnson (2005).”
What Do Recurrent Neural Network Grammars Learn About Syntax? (2016, background): “with each NT(X) capturing a left parenthesis and X nonterminal, and each REDUCE capturing a right parenthesis. Model F 1 Petrov and Klein (2007) 90.1 Shindo et al. (2012) 91.1 Zhu et al. (2013)y 91.3 McClosky et al. (2006)y 92.1 Vinyals et al. (2015)y 92.1 Choe and Charniak (2016) 92.6 Choe and Char”
Neural Generation Meets Real People: Towards Emotionally Engaging Mixed-Initiative Conversations (2020, methodology): “To improve performance on our data, we experimented with self-training (McClosky et al., 2006).”