Can Neural Machine Translation be Improved with User Feedback?

By Julia Kreutzer, Shahram Khadivi, Evgeny Matusov, Stefan Riezler. 2018

101 citations recorded by the source corpus.

We present the first real-world application of methods for improving neural machine translation (NMT) with human reinforcement, based on explicit and implicit user feedback collected on the eBay e-commerce platform. Previous work has been confined to simulation experiments, whereas in this paper we work with real logged feedback for offline bandit learning of NMT parameters. We conduct a thorough analysis of the available explicit user judgments—five-star ratings of translation quality—and show that they are not reliable enough to yield significant improvements in bandit learning. In contrast, we successfully utilize implicit task-based feedback collected in a cross-lingual search task to improve task-specific and machine translation quality metrics.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Training language models to follow instructions with human feedback (2022, background): “, 2019), translation (Kreutzer et al., 2018; Bahdanau et al., 2016), semantic parsing (Lawrence and Riezler, 2018), story generation (Zhou and Xu, 2020), review generation (Cho et al.”
Learning to summarize from human feedback (2020, background): “Human feedback has also been used as a reward to train models in other domains such as dialogue [25, 68, 21], translation [32, 1], semantic parsing [34], story generation [72], review generation [7], and evidence extraction [46].”
Recursively Summarizing Books with Human Feedback (2021, background): “, 2019), translation (Kreutzer et al., 2018; Bahdanau et al., 2016), semantic parsing (Lawrence and Riezler, 2018), story generation (Zhou and Xu, 2020), review generation (Cho et al.”
Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning (2018, background): “The first deployment of bandit NMT in an e-commerce translation scenario conjectured lacking reliability of user judgments as the reason for disappointing results when learning from 148k user-generated 5-star ratings for around 70k product title translations (Kreutzer et al., 2018).”
Humans or LLMs as the Judge? A Study on Judgement Bias (2024, methodology): “The collected feedback can be used to improve model performance (Kreutzer et al., 2018; Zhou and Xu, 2020; Leike et al., 2018; Ziegler et al., 2019; Stiennon et al., 2020; Böhm et al., 2019; Ouyang et al., 2022; Christiano et al., 2023) or to serve as an indicator of output quality as in Chatbot…”
AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback (2023, methodology): “RL research applied to NLP has a long history [75, 62, 27, 49, 42, 30, 29, 53, 61], and we expect future work in this direction to benefit from the ideas and artifacts in AlpacaFarm.”
Offline Reinforcement Learning from Human Feedback in Real-World Sequence-to-Sequence Tasks (2020, methodology): “2 Furthermore, interpreting feedback wrongly (e.g., through incorrect credit assignment (Bah-danau et al., 2017)), or receiving misleading feed-back (Nguyen et al., 2017; Kreutzer et al., 2018a), could easily push the policy into less favorable conditions.”
Preference-based interactive multi-document summarisation (2019, methodology): “Interactive Natural Language Processing (NLP) approaches that put the human in the loop gained increasing research interests recently (Amershi et al. 2014; Gurevych et al. 2018; Kreutzer et al. 2018a).”
ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation (2023, methodology): “Thus human feedback has been utilized to align model performance with intent in various language applications [1, 20, 33, 59] via training an RM [32, 8, 53, 19, 22] to learn human preference.”
Aligning Text-to-Image Models using Human Feedback (2023, methodology): “Human feedback has been used to improve various AI systems, from translation (Bahdanau et al., 2016; Kreutzer et al., 2018), to web question-answering (Nakano et al.”
Learning from Chunk-based Feedback in Neural Machine Translation (2018, background): “In (Kreutzer et al., 2018), the authors investigate the reliability and validity of real user ratings by re-evaluating ﬁve-star ratings by three independent human anno-tators, however the inter-annotator agreement be-tween experts was relatively low and no correlation to the averaged user rating…”
The rise and potential of large language model based agents: a survey (2025, unknown): “However, the results of Kreutzer et al. [354] suggest that such multi-level artiﬁcial ratings method might be ineﬃcient or less reliable.”