Discriminative Language Modeling with Conditional Random Fields and the Perceptron Algorithm

By Brian Roark, Murat Saraclar, Michael Collins, Mark Johnson. 2004

187 citations recorded by the source corpus.

This paper describes discriminative language modeling for a large vocabulary speech recognition task. We contrast two parameter estimation methods: the perceptron algorithm, and a method based on conditional random fields (CRFs). The models are encoded as deterministic weighted finite state automata, and are applied by intersecting the automata with word-lattices that are the output from a baseline recognizer. The perceptron algorithm has the benefit of automatically selecting a relatively small feature set in just a couple of passes over the training data. However, using the feature set output from the perceptron algorithm (initialized with their weights), CRF training provides an additional 0.5% reduction in word error rate, for a total 1.8% absolute reduction from the baseline of 39.2%.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Masked Language Model Scoring (2019, background): “Previous works (Roark et al., 2004; Huang et al., 2018) have explored training language models that directly optimize for a downstream metric (WER, BLEU).”
Domain Adaptation with Structural Correspondence Learning (2006, background): “Furthermore, end-to-end systems like speech recognizers (Roark et al., 2004) and automatic translators (Och, 2003) use increasingly sophisticated discriminative models, which generalize well to new data that is drawn from the same distribution as the training data.”
Learning Trans-Dimensional Random Fields with Applications to Language Modeling (2018, background): “n-gram LMs [6], neural network (NN) LMs [7], [8], [9] and conditional maximum entropy LMs [10], [11], [12].”
Random forests and the data sparseness problem in language modeling (2007, background): “Neural network language models and discriminative language models can achieve superior performance in reducing the perplexity and word error rate (Bengio et al., 2001; Schwenk and Gauvain, 2002; Emami et al., 2003; Roark et al., 2004a,b).”
An End-to-End Discriminative Approach to Machine Translation (2006, methodology): “Lexical language model features have been exploited successfully in discriminative language modeling to improve speech recognition performance (Roark et al., 2004).”
Efficient Inference in Large Conditional Random Fields (2006, methodology): “Other approximate inference techniques have been used to reduce CRF training time, such as beam-search during sum-product or max-product inference [16], or using the voted perceptron algorithm [17].”
A segmental CRF approach to large vocabulary continuous speech recognition (2009, unknown): “In the second approach, we jointly (and discriminatively) train the acoustic and language models - see also [20], [21].”
Learning Phonology With Substantive Bias: An Experimental and Computational Study of Velar Palatalization (2006, background): “Lafferty et al. (2001) introduce a general framework, referred to as conditional random fields (CRF), and apply it to the problem of labeling sequences (see also Gregory & Altun, 2004; Mc- Callum 2003; Roark et al., 2004; Sha & Pereira, 2003; among others).”
Trans-dimensional Random Fields for Language Modeling (2015, background): “The conditional ME models (Khudanpur and Wu, 2000; Roark et al., 2004) suffer from the expensive computation of local normalization factors.”