Overview of the IWSLT 2017 Evaluation Campaign

By Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Niehues Jan, Stüker Sebastian, Sudoh Katsuitho, Yoshino Koichiro, Federmann Christian. 2017

245 citations recorded by the source corpus.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Massively Multilingual Neural Machine Translation (2019, background): “Previous works on multilingual NMT typically trained models with up to 7 languages (Dong et al., 2015; Fi-rat et al., 2016b; Ha et al., 2016; Johnson et al., 2017; Gu et al., 2018) and up to 20 trained directions (Cettolo et al., 2017) simultaneously.”
A Call for Clarity in Reporting BLEU Scores (2018, background): “…its own metric-internal preprocessing, and produces the same values as WMT; • it automatically downloads and stores WMT (2008–2018) and IWSLT 2017 (Cettolo et al., 2017) test sets, obviating the need for the user to handle the references at all; and • it produces a short version string that…”
Beyond English-Centric Multilingual Machine Translation (2020, positive): “We use data for 4 languages (Cettolo et al., 2017).”
Cross-lingual Retrieval for Iterative Self-Supervised Training (2020, methodology): “Figure 2 shows the sentence retrieval accuracy of an mBART model that are ﬁnetuned with Japanese-English in the IWSLT17 parallel dataset ( 223 , 000 sentence pairs) [3].”
Achieving Human Parity on Automatic Chinese to English News Translation (2018, methodology): “4Results from both source-based and reference-based direct assessment collected for IWSLT17 [7] show that annotators assign higher scores in the source-based scenario and that they are more strict with their scoring in the reference-based scenario.”
Training Tips for the Transformer Model (2018, neutral): “the most recent evaluation campaigns (Bojar et al., 2017a; Cettolo et al., 2017).”
Contextual Parameter Generation for Universal Neural Machine Translation (2018, neutral): “(7)Note that, our results for IWSLT-17 are not comparable to those of the official challenge report (Cettolo et al., 2017), as we use less training data, a smaller baseline model, and our evaluation pipeline potentially differs.”
It’s not a Non-Issue: Negation as a Source of Error in Machine Translation (2020, background): “…of whether the improvements, as measured by evaluation campaigns such as WMT (Bo-jar et al., 2017, 2018a, 2019, et alia ) or IWSLT shared tasks (Cettolo et al., 2017; Niehues et al., 2018, et alia ) or other established benchmarks, do in fact mean better quality, especially with regards to the…”
Improving Zero-shot Translation with Language-Independent Constraints (2019, methodology): “The success of this method is shown by significant gains on zero-shot translation quality in the standard IWSLT 2017 multilingual benchmark (Cettolo et al., 2017).”
Findings of the WMT 2020 Shared Task on Automatic Post-Editing (2020, methodology): “Manual evaluation was conducted via source-based direct human assessment (Graham et al., 2013; Cettolo et al., 2017; Bojar et al., 2018).”
A Comparison of Transformer and Recurrent Neural Networks on Multilingual Neural Machine Translation (2018, methodology): “The experimental setting comprises seven languages; for each language pair, we use the ≈200,000 parallel sentences made publicly available by the IWSLT 2017 evaluation campaign (Cettolo et al., 2017), partitioned in training, development, and test sets.”
Bi-Directional Neural Machine Translation with Synthetic Parallel Data (2018, methodology): “Neural Machine Translation (NMT) has been rapidly adopted in industry as it consistently out-performs previous methods across domains and language pairs (Bojar et al., 2017; Cettolo et al., 2017).”
Effective Strategies in Zero-Shot Neural Machine Translation (2017, methodology): “We participated to this year’s IWSLT zero-shot tasks for German!Dutch and German!Romanian[12].”
Findings of the 2021 Conference on Machine Translation (WMT21) (2021, methodology): “We evaluated the overall quality of the MT and PE output using source-based direct assessment (Graham et al., 2013; Cettolo et al., 2017; Bojar et al., 2018b).”
PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts (2023, unknown): “• IWSLT 2017 (Cettolo et al., 2017) The IWSLT 2017 dataset (International Workshop on Spoken Language Translation 2017) is a collection of multilingual, multi-domain parallel text data specifically designed for evaluating spoken language translation systems.”
FINDINGS OF THE IWSLT 2020 EVALUATION CAMPAIGN (2020, unknown): “…conference (Akiba et al., 2004; Eck and Hori, 2005; Paul, 2006; Fordyce, 2007; Paul, 2008, 2009; Paul et al., 2010; Federico et al., 2011, 2012; Cettolo et al., 2013, 2014, 2015, 2016, 2017; Niehues et al., 2018, 2019) for the study, development and evaluation of spoken language translation…”
Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation (2018, methodology): “Experimental Unit Machine translation evaluation is predominantly performed on single sentences, presented to raters in random order (e. g., Bojar et al., 2017; Cettolo et al., 2017).”