Thinking Slow about Latency Evaluation for Simultaneous Machine Translation
By Colin Cherry, George F. Foster. 2019
54 citations recorded by the source corpus.
Simultaneous machine translation attempts to translate a source sentence before it is finished being spoken, with applications to translation of spoken language for live streaming and conversation. Since simultaneous systems trade quality to reduce latency, having an effective and interpretable latency metric is crucial. We introduce a variant of the recently proposed Average Lagging (AL) metric, which we call Differentiable Average Lagging (DAL). It distinguishes itself by being differentiable and internally consistent to its underlying mathematical model.
Subfields: Artificial Intelligence, Evaluation & Benchmarks, Machine Translation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Juan Pino
- Changhan Wang
- Jiatao Gu
- Graham Neubig
- Yonatan Belinkov
- Philipp Koehn
- Felix Stahlberg
- Orhan Firat
In Evaluation & Benchmarks
- Marco Turchi
- Matteo Negri
- Marcello Federico
- Ondrej Bojar
- Jan Niehues
- Christian Federmann
- Nadir Durrani
- Colin Cherry
In Machine Translation
Citations in the corpus
- Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq (2020, methodology): “It also integrates SIMULEVAL (Ma et al., 2020a) for simultaneous ST/MT metrics such as AL (average lagging) (Ma et al., 2019) and DAL (differentiable average Lagging) (Cherry and Foster, 2019).”
- Monotonic Infinite Lookback Attention for Simultaneous Machine Translation (2019, background): “Cherry and Foster (2019) provide further motivation and analysis for DAL, alongside several examples of cases where DAL yields more intuitive results than AL.”
- SIMULEVAL: An Evaluation Toolkit for Simultaneous Translation (2020, background): “(4): where A minimum delay prevent DAL recovering from lagging once it has been incurred.”
- FINDINGS OF THE IWSLT 2020 EVALUATION CAMPAIGN (2020, methodology): “Latency was evaluated with the recently developed metrics for simultaneous machine translation including average proportion (AP), average lagging (AL) and differentiable average lagging (DAL) (Cherry and Foster, 2019).”