compare-mt: A Tool for Holistic Comparison of Language Generation Systems

By Graham Neubig, Zi-Yi Dou, Junjie Hu, Paul Michel, Danish Pruthi, Xinyi Wang. 2019

118 citations recorded by the source corpus.

In this paper, we describe compare-mt, a tool for holistic analysis and comparison of the results of systems for language generation tasks such as machine translation. The main goal of the tool is to give the user a high-level and coherent view of the salient differences between systems that can then be used to guide further analysis or system improvement. It implements a number of tools to do so, such as analysis of accuracy of generation of particular types of words, bucketed histograms of sentence accuracies or counts based on salient characteristics, and extraction of characteristic n-grams for each system. It also has a number of advanced features such as use of linguistic labels, source side data, or comparison of log likelihoods for probabilistic models, and also aims to be easily extensible by users to new types of analysis. compare-mt is a pure-Python open source package, that has already proven useful to generate analyses that have been used in our published papers. Demo Video: https://youtu.be/NyJEQT7t2CA

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Mask-Predict: Parallel Decoding of Conditional Masked Language Models (2019, methodology): “To determine whether this is the case, we use compare-mt (Neubig et al., 2019) to bucket the evaluation data by target sentence length and compute the performance with different values of T .”
ExplainaBoard: An Explainable Leaderboard for NLP (2021, background): “Interpretable evaluation (Popovi´c and Ney, 2011; Stymne, 2011; Neubig et al., 2019; Fu et al., 2020a), is a research area that considers methods that break down the holistic performance of each system into different interpretable groups.”
Decoding and Diversity in Machine Translation (2020, methodology): “We examine distributional differences including correct prediction rates of female pronouns, word frequency, sentence length, and the distribution of sentence-level BLEU scores between the two systems using compare-mt (Neubig et al., 2019).”
Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation (2020, methodology): “We use the compare-mt toolkit by Neubig et al. (2019).”
Beyond BLEU:Training Neural Machine Translation with Semantic Similarity (2019, methodology): “The seminal work on training machine translation systems to optimize particular evaluation measures was performed by Och (2003), who introduced minimum error rate training (MERT) and used it to optimize several different metrics in statistical MT (SMT).”
VizSeq: a visual analysis toolkit for text generation tasks (2019, methodology): “…of various metrics or demonstrating examples with sentence-level scores in an integrated user interface: ibleu (Madnani, 2011), MTEval 1, MT-ComparEval (Klejch et al., 2015), nlg-eval (Sharma et al., 2017), Vis-Eval Metric Viewer (Steele and Specia, 2018), comparemt (Neubig et al., 2019), etc.”
Interpretable Multi-dataset Evaluation for Named Entity Recognition (2020, background): “Bucketing is an operation that breaks down the holistic performance into different categories (Neubig et al., 2019; Fu et al., 2020).”
Word Alignment by Fine-tuning Embeddings on Parallel Corpora (2021, background): “…(Liu et al., 2016), incorporating lexicons during decoding (Arthur et al., 2016), domain adaptation (Hu et al., 2019), unsupervised MT (Ren et al., 2020) and automatic evaluation or analysis of translation models (Bau et al., 2018; Stanovsky et al., 2019; Neubig et al., 2019; Wang et al., 2020).”
Non-Autoregressive Machine Translation with Latent Alignments (2020, methodology): “We use the compare-mt (Neubig et al., 2019) package to bucket test set examples based on target sentence length, and compute BLEU score for each bucket using different number of decoding iterations.”
The statistical advantage of automatic NLG metrics at the system level (2021, methodology): “This includes the traditional ROUGE-4 (Lin, 2004) summarization metric, its extension ROUGE-WE (Ng and Abrecht, 2015), and METEOR (Lavie and Agar-wal, 2007).”