Bandit Structured Prediction for Neural Sequence-to-Sequence Learning

By Julia Kreutzer, Artem Sokolov, Stefan Riezler. 2017

56 citations recorded by the source corpus.

Bandit structured prediction describes a stochastic optimization framework where learning is performed from partial feedback. This feedback is received in the form of a task loss evaluation to a predicted output structure, without having access to gold standard structures. We advance this framework by lifting linear bandit learning to neural sequence-to-sequence learning problems using attention-based recurrent neural networks. Furthermore, we show how to incorporate control variates into our learning algorithms for variance reduction and improved generalization. We present an evaluation on a neural machine translation task that shows improvements of up to 5.89 BLEU points for domain adaptation from simulated bandit feedback.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Better Rewards Yield Better Summaries: Learning to Summarise Without References (2019, methodology): “Our method only needs human ratings on some generated summaries, also known as the bandit feedback (Kreutzer et al., 2017), to learn the reward function.”
Can Neural Machine Translation be Improved with User Feedback? (2018, background): “Nguyen et al., 2017; Kreutzer et al., 2017; Bahdanau et al., 2017).”
Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning (2018, background): “Our aim is to show that successful learning from simulated bandit feedback (Sokolov et al., 2016b; Kreutzer et al., 2017; Nguyen et al., 2017; Lawrence et al., 2017) does in fact carry over to learning from actual human bandit feedback.”
Preference-based interactive multi-document summarisation (2019, methodology): “5); 6 end Output: y = arg maxy wN · φ(y|x) Algorithm 1: Adaptation of SPPI (Kreutzer et al. 2017, Alg.”
A Shared Task on Bandit Learning for Machine Translation (2017, methodology): “The goal of this shared task is to evaluate existing algorithms for learning MT systems from weak feedback (Sokolov et al., 2015, 2016a; Kreutzer et al., 2017) on real-world data and compare them to new algorithms, with a focus on performing online learning efﬁciently and effectively from bandit…”
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning (2018, methodology): “Preference-based SPPI has been applied to multiple NLP applications, including text classiﬁcation, chunking and machine translation (Sokolov et al., 2016a; Kreutzer et al., 2017).”
Learning from Chunk-based Feedback in Neural Machine Translation (2018, background): “Integrating user ratings in NMT has been studied in (Kreutzer et al., 2017), who view this as a bandit structured prediction task.”
Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translation (2021, methodology): “(2020) missed a few crucial aspects of RL that have led to empirical success in previous works: First, variance reduction techniques such as the average reward baseline were already proposed with the original Policy Gradient by Williams (1992), and proved effective for NMT (Kreutzer et al., 2017; Ng”
A Reinforcement Learning Approach to Interactive-Predictive Neural Machine Translation (2018, methodology): “…operates in a fully supervised learning scenario, where rewards are simulated in comparison to references with smoothed and length-rescaled BLEU, Kreutzer et al. (2017) and Nguyen et al. (2017) limit the setup to sentence-level bandit feedback, i.e. only one feedback is obtained for one…”
Counterfactual Learning for Machine Translation: Degeneracies and Solutions (2017, background): “Recent research has proposed bandit structured prediction (Sokolov et al., 2016; Kreutzer et al., 2017; Nguyen et al., 2017) for online learning of machine translation from weak user feedback to predicted translations, instead of from costly manually created reference translations.”
Counterfactual Learning from Bandit Feedback under Deterministic Logging : A Case Study in Statistical Machine Translation (2017, methodology): “learning protocol using the terminology of bandit structured prediction (Sokolov et al., 2016; Kreutzer et al., 2017), where at each round, a system (corresponding to a policy in RL terms) makes a prediction (also called action in RL, or pulling an arm of a bandit), and receives a reward, which is u”