DiaBLa: a corpus of bilingual spontaneous written dialogues for machine translation

By Rachel Bawden, Sophie Rosset, Thomas Lavergne, Éric Bilinski. 2019

18 citations recorded by the source corpus.

We present a new English–French dataset for the evaluation of Machine Translation (MT) for informal, written bilingual dialogue. The test set contains 144 spontaneous dialogues (5700+ sentences) between native English and French speakers, mediated by one of two neural MT systems in a range of role-play settings. The dialogues are accompanied by fine-grained sentence-level judgments of MT quality, produced by the dialogue participants themselves, as well as by manually normalised versions and reference translations produced a posteriori. The motivation for the corpus is twofold: to provide (i) a unique resource for evaluating MT models, and (ii) a corpus for the analysis of MT-mediated communication. We provide an initial analysis of the corpus to confirm that the participants’ judgments reveal perceptible differences in MT quality between the two MT systems used.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, methodology): “We evaluate using the sacrebleu (Post, 2018) implementation of BLEU (Papineni et al., 2002), using default tokenisation for WMT and DiaBLa and spm-flores-101 for Flores.30 We use greedy decoding with generation proceeding until the EOS token, or additionally \n###\n for the 1-shot case.”
A Survey of Large Language Models (2023, unknown): “Conditional Text Generation WMT’14,16,19,20,21,22 [271–276], Flores-101 [277], DiaBLa [278], CNN/DailyMail [279], XSum [280], WikiLingua [281], OpenDialKG [282] SuperGLUE [283], MMLU [284], BIG-bench Hard [285], CLUE [286]”
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM (2023, unknown): “Few-shot examples are randomly taken from the data splits according to availability (train for WMT, dev for Flores-101 and test for DiaBLa).”