A Systematic Comparison of Various Statistical Alignment Models

By Franz Joseph Och, Hermann Ney. 2003

4,656 citations recorded by the source corpus.

We present and compare various methods for computing word alignments using statistical or heuristic models. We consider the five alignment models presented in Brown, Della Pietra, Della Pietra, and Mercer (1993), the hidden Markov alignment model, smoothing techniques, and refinements. These statistical models are compared with two heuristic models based on the Dice coefficient. We present different methods for combining word alignments to perform a symmetrization of directed statistical alignment models. As evaluation criterion, we use the quality of the resulting Viterbi alignment compared to a manually produced reference alignment. We evaluate the models on the German-English Verbmobil task and the French-English Hansards task. We perform a detailed analysis of various design decisions of our statistical alignment system and evaluate these on training corpora of various sizes. An important result is that refined alignment models with a first-order dependence and a fertility model yield significantly better results than simple heuristic models. In the Appendix, we present an efficient training algorithm for the alignment models presented.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Findings of the 2016 Conference on Machine Translation (2016, methodology): “To produce these features, GIZA++ (Och and Ney, 2003) was used for word alignment and Multivec (Berard et al., 2016) was used for the bilingual model, which jointly learns distributed representations for source and target languages using a parallel corpus.”
Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics (2013, methodology): “We then use Giza++ (Och & Ney, 2003) to train IBM alignment models 1–2 (Brown et al., 1993) over all pairs of noun or verb captions of the same image to obtain two sets of translation probabilities, one over nouns (P n (·|w)) and one over verbs (P v (·|w)).”
SIGMa: simple greedy matching for aligning large knowledge bases (2012, background): “Another approach to alignment arises from the word alignment problem in natural language processing [23], which has been formulated as a maximum weighted bipartite matching problem [29] (thus exploiting the 1–1 matching structure).”
Neural Approaches to Conversational AI (2018, critical): “to signiﬁcantly improve over RNNs thanks to its ability to repeat proper nouns and other words of the input. 4Ritter et al. (2011) also found that alignment produced by an off-the-shelf word aligner (Och and Ney, 2003) produced alignments of poor quality, and an extension of their work with attentio”
Neural Responding Machine for Short-Text Conversation (2015, methodology): “The main modification is in replacing the standard GIZA++ word alignment model (Och and Ney, 2003) with a new phrase-pair selection method, in which all the 6we use the default similarity function of Lucene 7 possible phrase-pairs in the training data are considered and their associated…”
Parallel Data, Tools and Interfaces in OPUS (2012, methodology): “We applied heavy ﬁltering using probability thresholds, frequency thresholds and string patterns.”
Modeling Coverage for Neural Machine Translation (2016, methodology): “To better estimate the quality of the soft alignment probabilities generated by NMT, we propose a variant of AER, naming SAER : where A is a candidate alignment, and S and P are the sets of sure and possible links in the reference alignment respectively ( S ⊆ P ).”
Unsupervised Construction of Large Paraphrase Corpora: Exploiting Massively Parallel News Sources (2004, methodology): “To compute Precision, Recall, and Alignment Error Rate (AER) for the twin datasets, we used exactly the formulae listed in Och & Ney (2003).”
(Meta-) Evaluation of Machine Translation (2007, methodology): “The word alignments were created with Giza++ (Och and Ney, 2003) applied to a parallel corpus containing 200,000 sentence pairs of the training data, plus sets of 4,007 sentence pairs created by pairing the test sentences with the reference translations, and the test sentences paired with each of th”
Learning for Semantic Parsing with Statistical Machine Translation (2006, methodology): “In this work, we use the GIZA++ implementation (Och and Ney, 2003) of IBM Model 5 (Brown et al.”
Decoding with Large-Scale Neural Language Models Improves Translation (2013, methodology): “We ran MERT on the development data, which was the NIST 2003 test data, and tested on the NIST 2004– 2006 test data.”
Alignment by Agreement (2006, methodology): “The distortion parameterspd(aj = i′ | aj − = i) depend on the particular model (we write aj = 0 to denote the event that thej-th French word 2The dependence onaj − can in fact be implemented as a first-order HMM (see Och and Ney (2003)). is null-aligned): pd(aj =0 | aj − = i) = p0 pd(aj = i ′ 6= 0 |”
BilBOWA: Fast Bilingual Distributed Representations without Word Alignments (2014, methodology): “…scores were computed on sequences of words, but in Mikolov et al. [15] this idea was taken one step further and successfully applied to bag-of-word representations of contexts in their continuous bag-of-words (CBOW) and skipgram models trained using the negative sampling training objective.”
Information Extraction over Structured Data: Question Answering with Freebase (2014, methodology): “, 1993) to estimate the translation probability with GIZA++ (Och and Ney, 2003).”
Unsupervised Statistical Machine Translation (2018, methodology): “ments in both directions from a parallel corpus, symmetrizes these alignments using different heuristics (Och and Ney, 2003), extracts the set of consistent phrase pairs, and scores them based on frequency counts.”
Training a Multilingual Sportscaster: Using Perceptual Context to Learn Language (2014, methodology): “First, an SMT word alignment system, GIZA++ (Och & Ney, 2003; Brown, Della Pietra, Della Pietra, & Mercer, 1993), is used to acquire a bilingual lexicon consisting of NL substrings coupled return The parsing score 31: end function with their translations in the target MRL.”