Counterfactual Learning from Bandit Feedback under Deterministic Logging : A Case Study in Statistical Machine Translation

By Carolin (Haas) Lawrence, Artem Sokolov, Stefan Riezler. 2017

33 citations recorded by the source corpus.

The goal of counterfactual learning for statistical machine translation (SMT) is to optimize a target SMT system from logged data that consist of user feedback to translations that were predicted by another, historic SMT system. A challenge arises by the fact that risk-averse commercial SMT systems deterministically log the most probable translation. The lack of sufficient exploration of the SMT output space seemingly contradicts the theoretical requirements for counterfactual learning. We show that counterfactual learning from deterministic bandit logs is possible nevertheless by smoothing out deterministic components in learning. This can be achieved by additive and multiplicative control variates that avoid degenerate behavior in empirical risk minimization. Our simulation experiments show improvements of up to 2 BLEU points by counterfactual learning from deterministic bandit feedback.

Subfields: Artificial Intelligence, Management Science and Operations Research, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback (2018, methodology): “This results in a deterministic propensity matching (DPM) objec-tive (Lawrence et al., 2017b), without the possibility to correct the sampling bias of the logging policy by inverse propensity scoring (Rosenbaum and Rubin, 1983): This objective can show degenerate behavior in that it overﬁts to the…”
Can Neural Machine Translation be Improved with User Feedback? (2018, methodology): “…signal for response prediction in online advertising (Bottou et al., 2013), bandit learning for machine translation has so far been restricted to simulation experiments (Sokolov et al., 2016b; Lawrence et al., 2017b; ∗The work for this paper was done while the first author was an intern at eBay.”
Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning (2018, methodology): “Our aim is to show that successful learning from simulated bandit feedback (Sokolov et al., 2016b; Kreutzer et al., 2017; Nguyen et al., 2017; Lawrence et al., 2017) does in fact carry over to learning from actual human bandit feedback.”
A Shared Task on Bandit Learning for Machine Translation (2017, background): “A desideratum for a future installment of this shared task is the option to perform ofﬂine learning from bandit feedback (Lawrence et al., 2017), thus allowing a more lightweight infrastructure, and opening the task to (mini)batch learning techniques that are more standard in the ﬁeld of machine…”
MultiVerse: Causal Reasoning using Importance Sampling in Probabilistic Programming (2019, background): “, 2018), translation (Lawrence et al., 2017), and reinforcement learning (Foerster et al.”
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning (2020, background): “, 2017), and improving the performance of NLP tasks using counterfactual learning (Lawrence et al., 2017; Lawrence and Riezler, 2018), it remains a major research challenge.”
Learning from Chunk-based Feedback in Neural Machine Translation (2018, methodology): “Lawrence et al. (2017) have proposed a method to leverage user feedback that is available only for logged translated data for a phrase-based statistical machine translation system.”
Offline Reinforcement Learning from Human Feedback in Real-World Sequence-to-Sequence Tasks (2020, methodology): “This has been explored in the context of machine translation (Lawrence et al., 2017b), utilizing the Deterministic Propensity Matching (DPM) objective”
Counterfactual Learning for Machine Translation: Degeneracies and Solutions (2017, methodology): “…degeneracies of the standard inverse propensity scoring technique (Rosenbaum and Rubin, 1983) and its reweighted variant (Kong, 1992) under stochastic and deterministic logging, with the goal of a clearer understanding of the effectiveness of the techniques proposed in Lawrence et al. (2017).”