Overview of the Second BUCC Shared Task: Spotting Parallel Sentences in Comparable Corpora

By Pierre Zweigenbaum, Serge Sharoff, Reinhard Rapp. 2017

142 citations recorded by the source corpus.

This paper presents the BUCC 2017 shared task on parallel sentence extraction from comparable corpora. It recalls the design of the datasets, presents their final construction and statistics and the methods used to evaluate system results. 13 runs were submitted to the shared task by 4 teams, covering three of the four proposed language pairs: French-English (7 runs), German-English (3 runs), and Chinese-English (3 runs). The best F-scores as measured against the gold standard were 0.84 (German-English), 0.80 (French-English), and 0.43 (Chinese-English). Because of the design of the dataset, in which not all gold parallel sentence pairs are known, these are only minimum values. We examined manually a small sample of the false negative sentence pairs for the most precise French-English runs and estimated the number of parallel sentence pairs not yet in the provided gold standard. Adding them to the gold standard leads to revised estimates for the French-English F-scores of at most +1.5pt. This suggests that the BUCC 2017 datasets provide a reasonable approximate evaluation of the parallel sentence spotting task.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Filtering and Mining Parallel Data in a Joint Multilingual Space (2018, background): “Since 2017, the workshop on Building and Using Comparable Corpora (BUCC) is organizing a shared task to evaluate the performance of approaches to mine for parallel sentences in comparable corpora (Zweigenbaum et al., 2018).”
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization (2020, background): “BUCC The goal of the second and third shared task of the workshop on Building and Using Parallel Corpora (Zweigenbaum et al., 2017; 2018) is to extract parallel sentences from a comparable corpus between English and four other languages.”
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond (2018, methodology): “We use this method to evaluate our sentence embeddings on the BUCC mining task (Zweigenbaum et al., 2017, 2018), using exact same hyperparameters as Artetxe and Schwenk (2018).”
Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings (2018, methodology): “In order to perform an intrinsic evaluation of our method for parallel corpus mining, we use the standard dataset from the 2017 shared task of the workshop on Building and Using Comparable Corpora (BUCC) (Zweigenbaum et al., 2017).”
Pre-training via Paraphrasing (2020, methodology): “We report performance on BUCC2018 [Zweigenbaum et al., 2018] and Tatoeba [Artetxe and Schwenk, 2019].”
Making Monolingual Sentence Embeddings Multilingual Using Knowledge Distillation (2020, methodology): “We use the dataset from the BUCC mining task (Zweigenbaum et al., 2017, 2018), with the goal of extracting parallel sentences between an English corpus and four other languages: German, French, Russian, and Chinese.”
CCMatrix: Mining Billions of High-Quality Parallel Sentences on the Web (2019, methodology): “This strategy was initially motivated by an evaluation on the BUCC corpus (Zweigenbaum et al., 2018), for which the reference alignments were known to be strictly 1:1.”
An Empirical Analysis of NMT-Derived Interlingual Embeddings and Their Use in Parallel Sentence Identification (2017, methodology): “We use the de–en and fr–en training corpora provided for the shared task on identifying parallel sentences in comparable corpora at BUCC 2017 [43].”
MTEB: Massive Text Embedding Benchmark (2022, background): “BUCC (Zweigenbaum et al., 2016, 2017, 2018) BUCC provides big set of sentences (∼ 10-70k each) for English, French, Russian, German and Chinese, along with associated pairs annotation.”
VECO: Variable Encoder-decoder Pre-training for Cross-lingual Understanding and Generation (2020, unknown): “BUCC 2018 (Zweigenbaum et al., 2018), Tatoeba (Artetxe & Schwenk, 2019).”
Extracting Parallel Sentences from Comparable Corpora with STACC Variants (2018, methodology): “We selected as our baseline the weighted variant of the approach (Azpeitia et al., 2017), which proved highly successful on the BUCC 2017 shared task (Zweigenbaum et al., 2017), and applied the approach to all four language pairs in the 2018 task.”
Unsupervised Parallel Sentence Extraction with Parallel Segment Detection Helps Machine Translation (2019, methodology): “First, we evaluate the accuracy of the mining approach on the BUCC 2017 shared task data (Zweigenbaum et al., 2017).”
Unsupervised Multilingual Sentence Embeddings for Parallel Corpus Mining (2020, methodology): “When comparing our method to related work, it must be noted that the XLM model was pretrained on Wikipedia and therefore has seen the monolingual BUCC sentences during training.”
FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding (2020, methodology): “Cross-lingual Sentence Retrieval includes two tasks: BUCC (Zweigenbaum, Sharoff, and Rapp 2018) and Tatoeba (Artetxe and Schwenk 2019).”
Simple and Effective Paraphrastic Similarity from Parallel Translations (2019, methodology): “Second, we compare our best model, SP, on two semantic cross-lingual tasks: the 2017 SemEval STS task (Cer et al., 2017) which consists of monolingual and cross-lingual datasets and the 2018 Building and Using Parallel Corpora (BUCC) shared bitext mining task (Zweigenbaum et al., 2018).”
Parallel Sentence Mining by Constrained Decoding (2020, methodology): “The results from Building and Using Comparable Corpora (BUCC) shared task show that direct sentence alignment can be done by sentence-level lexical comparison, neural comparison or a combination of the two (Zweigenbaum et al., 2017, 2018).”
A Bilingual Generative Transformer for Semantic Sentence Embedding (2019, background): “(2012)), mining bitext (Zweigenbaum et al., 2018), and paraphrase identification (Dolan et al.”