Human Effort and Machine Learnability in Computer Aided Translation

By Spence Green, Sida I. Wang, Jason Chuang, Jeffrey Heer, Sebastian Schuster, Christopher D. Manning. 2014

72 citations recorded by the source corpus.

Analyses of computer aided translation typically focus on either frontend interfaces and human effort, or backend translation and machine learnability of corrections. However, this distinction is artificial in practice since the frontend and backend must work in concert. We present the first holistic, quantitative evaluation of these issues by contrasting two assistive modes: postediting and interactive machine translation (MT). We describe a new translator interface, extensive modifications to a phrasebased MT system, and a novel objective function for re-tuning to human corrections. Evaluation with professional bilingual translators shows that post-edit is faster than interactive at the cost of translation quality for French-English and EnglishGerman. However, re-tuning the MT system to interactive output leads to larger, statistically significant reductions in HTER versus re-tuning to post-edit. Analysis shows that tuning directly to HTER results in fine-grained corrections to subsequent machine output.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning (2018, methodology): “Lastly, we investigate machine learnability of the overall NMT task, in the sense of Green et al. (2014) who posed the question of how well an MT system can be tuned on post-edits.”
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models (2021, background): “Not only can this decrease users’ creativity (Green et al., 2014), but it may bias their analysis process: as discussed in §7, Polyjuice generation is not exhaustive, and may favor some perturbation patterns over others in unpredictable ways.”
Neural Interactive Translation Prediction (2016, methodology): “In prior work, phrase-based machine translation systems have been used for interactive translation prediction, and suggestions were made either by re-decoding constrained by the prefix (Green et al., 2014) or by searching for the prefix in the original search graph (Och et al.”
Stochastic Structured Prediction under Bandit Feedback (2016, unknown): “This motivates the formalization of convex upper bounds on expec ted normalized loss as presented in [10].”
QuickEdit: Editing Text & Translations via Simple Delete Actions (2017, methodology): “Green et al. (2014) implement such a sys-tem relying on a phrase-based translation system.”
Visual Interaction with Deep Learning Models through Collaborative Semantic Inference (2019, background): “While previous work on interactive phrase-based machine translation showed promising results towards the goal of collaborative interfaces, the same techniques are not possible with deep learning-based approaches [28].”
Learning Structured Predictors from Bandit Feedback for Interactive NLP (2016, methodology): “For example, in interactive statistical machine translation (SMT), user feedback in form of post-edits of predicted translations is used for model adaptation (Bertoldi et al., 2014; Denkowski et al., 2014; Green et al., 2014).”
Assessing the Impact of Automated Suggestions on Decision Making: Domain Experts Mediate Model Errors but Take Less Initiative (2021, methodology): “When translators used an interface that displayed machine-suggested recommendations, they noted that they ceded agency and would conform to the machine’s recommended phrasing, even when it did not match their usual style [15].”
A Reinforcement Learning Approach to Interactive-Predictive Neural Machine Translation (2018, methodology): “The interactive-predictive translation paradigm reaches back to early approaches for IBM-type (Foster et al., 1997; Foster et al., 2002) and phrase-based machine translation (Barrachina et al., 2008; Green et al., 2014).”
QuickEdit: Editing Text & Translations by Crossing Words Out (2017, methodology): “Green et al. (2014) implement such a system relying on a phrase-based translation system.”
Sparse Sequence-to-Sequence Models (2019, background): “This has potential practical applications as a predictive translation system (Green et al., 2014), where the model’s support set serves as a list of candidate auto-completions at each time step.”