VizSeq: a visual analysis toolkit for text generation tasks

By Changhan Wang, Anirudh Jain, Danlu Chen, Jiatao Gu. 2019

30 citations recorded by the source corpus.

Automatic evaluation of text generation tasks (e.g. machine translation, text summarization, image captioning and video description) usually relies heavily on task-specific metrics, such as BLEU and ROUGE. They, however, are abstract numbers and are not perfectly aligned with human assessment. This suggests inspecting detailed examples as a complement to identify system error patterns. In this paper, we present VizSeq, a visual analysis toolkit for instance-level and corpus-level system evaluation on a wide variety of text generation tasks. It supports multimodal sources and multiple text references, providing visualization in Jupyter notebook or a web app interface. It can be used locally or deployed onto public servers for centralized data hosting and benchmarking. It covers most common n-gram based metrics accelerated with multiprocessing, and also provides latest embedding-based metrics such as BERTScore.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Generation, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Juan Pino
2. Changhan Wang
3. Wei-Ning Hsu
4. Yossi Adi
5. Ann Lee
6. Jiatao Gu
7. Peng-Jen Chen
8. Yun Tang

In Computer Vision and Pattern Recognition

1. Juan Pino
2. Jiatao Gu
3. Changhan Wang
4. Wei-Ning Hsu
5. Yossi Adi
6. Anne Wu
7. Xutai Ma
8. Peng-Jen Chen

In Natural Language Generation

1. Juan Pino
2. Anne Wu
3. Changhan Wang
4. Jiatao Gu
5. Adam Polyak

In Human-Computer Interaction

1. Juan Pino
2. Anne Wu
3. Changhan Wang
4. Jiatao Gu

Citations in the corpus

• Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq (2020, methodology): “It also has VizSeq (Wang et al., 2019) 13.7 High Lat.”
• fairseq Sˆ2: A Scalable and Integrable Speech Synthesis Toolkit (2021, methodology): “It also has VizSeq (Wang et al., 2019) integration for offline sequence-level error analysis,”
• Improving Cross-Lingual Transfer Learning for End-to-End Speech Recognition with Speech Translation (2020, methodology): “For ASR, we report character error rate (CER) on Chinese (no word segmentation) and word error rate (WER) on the other languages using VizSeq [37].”
• Self-Supervised Representations Improve End-to-End Speech Translation (2020, methodology): “For ASR, the reported word error rate (WER) has been obtained using VizSeq [31].”
• CoVoST 2: A Massively Multilingual Speech-to-Text Translation Corpus (2020, methodology): “For ASR, we report character error rate (CER) on Japanese and Chinese (no word segmentation) and word error rate (WER) on the other languages using VizSeq (Wang et al., 2019).”