SIMULEVAL: An Evaluation Toolkit for Simultaneous Translation

By Xutai Ma, Mohammad Javad Dousti, Changhan Wang, Jiatao Gu, Juan Pino. 2020

130 citations recorded by the source corpus.

Simultaneous translation on both text and speech focuses on a real-time and low-latency scenario where the model starts translating before reading the complete source input. Evaluating simultaneous translation models is more complex than offline models because the latency is another factor to consider in addition to translation quality. The research community, despite its growing focus on novel modeling approaches to simultaneous translation, currently lacks a universal evaluation procedure. Therefore, we present SimulEval, an easy-to-use and general evaluation toolkit for both simultaneous text and speech translation. A server-client scheme is introduced to create a simultaneous translation scenario, where the server sends source input and receives predictions for evaluation and the client executes customized policies. Given a policy, it automatically performs simultaneous decoding and collectively reports several popular latency metrics. We also adapt latency metrics from text simultaneous translation to the speech task. Additionally, SimulEval is equipped with a visualization interface to provide better understanding of the simultaneous decoding process of a system. SimulEval has already been extensively used for the IWSLT 2020 shared task on simultaneous speech translation. Code will be released upon publication.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Juan Pino
2. Changhan Wang
3. Ann Lee
4. Wei-Ning Hsu
5. Peng-Jen Chen
6. Yun Tang
7. Hongyu Gong
8. Philipp Koehn

In Evaluation & Benchmarks

1. Philipp Koehn
2. Kenneth Heafield
3. Juan Pino
4. Peng-Jen Chen
5. Changhan Wang
6. Hongyu Gong
7. Dmytro Okhonko
8. Ann Lee

In Machine Translation

1. Juan Pino
2. Changhan Wang
3. Yun Tang
4. Anne Wu
5. Xutai Ma
6. Ann Lee
7. Dmytro Okhonko
8. Wei-Ning Hsu

Citations in the corpus

• Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq (2020, methodology): “It also integrates SIMULEVAL (Ma et al., 2020a) for simultaneous ST/MT metrics such as AL (average lagging) (Ma et al.”
• Direct simultaneous speech to speech translation (2021, methodology): “However, such estimation is non-trivial, since inference time alignment is not differentiable during training.”