Aligning Sentences in Parallel Corpora

By Peter F. Brown, Jennifer C. Lai, Robert L. Mercer. 1991

In this paper we describe a statistical technique for aligning sentences with their translations in two parallel corpora. In addition to certain anchor points that are available in our data, the only information about the sentences that we use for calculating alignments is the number of tokens that they contain. Because we make no use of the lexical details of the sentence, the alignment computation is fast and therefore practical for application to very large collections of text. We have used this technique to align several million sentences in the English-French Hansard corpora and have achieved an accuracy in excess of 99% in a random selected set of 1000 sentence pairs that we checked by hand. We show that even without the benefit of anchor points the correlation between the lengths of aligned sentences is strong enough that we should expect to achieve an accuracy of between 96% and 97%. Thus, the technique may be applicable to a wider variety of texts than we have yet tried.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks (2020, background): “, [3], to the best of our knowledge, this work is the first that explores the idea for VLP.”
Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfer (2021, background): “The pivoting idea for alignment learning can date back to Brown et al. (1991).”
Parallel Sentence Mining by Constrained Decoding (2020, background): “With the assumption that matched documents are parallel (no cross-alignment), sentence alignment can be done by comparing sentence length in words (Brown et al., 1991) or characters (Gale and Church, 1993), which is then improved by adding lexical features (Varga et al., 2005).”
The Mathematics of Statistical Machine Translation: Parameter Estimation (1993, methodology): “Brown et al. (1990) assert, and Brown, Lai, and Mercer (1991) and Gale and Church (1991b) both show, that it is possible to obtain such aligned pairs of sentences without inspecting the words that the sentences contain.”
Fast and accurate sentence alignment of bilingual corpora (2002, methodology): “’s [3] sentence-length-based model to use Poisson distributions, rather than Gaussians, so that no hidden parameters need to be iteratively re-estimated.”