Stochastic Structured Prediction under Bandit Feedback

By Artem Sokolov, Julia Kreutzer, Stefan Riezler, Christopher Lo. 2016

31 citations recorded by the source corpus.

Stochastic structured prediction under bandit feedback follows a learning protocol where on each of a sequence of iterations, the learner receives an input, predicts an output structure, and receives partial feedback in form of a task loss evaluation of the predicted structure. We present applications of this learning scenario to convex and non-convex objectives for structured prediction and analyze them as stochastic first-order methods. We present an experimental evaluation on problems of natural language processing over exponential output spaces, and compare convergence speed across different objectives under the practical criterion of optimal task performance on development data and the optimization-theoretic criterion of minimal squared gradient norm. Best results under both criteria are obtained for a non-convex objective for pairwise preference learning under bandit feedback.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Can Neural Machine Translation be Improved with User Feedback? (2018, methodology): “…signal for response prediction in online advertising (Bottou et al., 2013), bandit learning for machine translation has so far been restricted to simulation experiments (Sokolov et al., 2016b; Lawrence et al., 2017b; ∗The work for this paper was done while the first author was an intern at eBay.”
Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning (2018, background): “Our aim is to show that successful learning from simulated bandit feedback (Sokolov et al., 2016b; Kreutzer et al., 2017; Nguyen et al., 2017; Lawrence et al., 2017) does in fact carry over to learning from actual human bandit feedback.”
Bandit Structured Prediction for Neural Sequence-to-Sequence Learning (2017, methodology): “In the following, we will present objectives from Sokolov et al. (2016b) transferred to neural models, and explain how they can be enhanced by con-trol variates.”
Offline Reinforcement Learning from Human Feedback in Real-World Sequence-to-Sequence Tasks (2020, background): “The majority of RL research in NLP has focused on learning from online feedback (Sokolov et al., 2016; He et al., 2016; Li et al., 2016; Bahdanau et al., 2017; Nguyen et al., 2017; Nogueira and Cho, 2017; Lam et al., 2018).”
Preference-based interactive multi-document summarisation (2019, methodology): “1 shows this general idea in comparison to the state-of-the-art preference-based interactive NLP paradigm, Structured Prediction from Partial Information (SPPI) (Sokolov et al. 2016b; Kreutzer et al. 2017).”
Counterfactual Learning for Machine Translation: Degeneracies and Solutions (2017, background): “Recent research has proposed bandit structured prediction (Sokolov et al., 2016; Kreutzer et al., 2017; Nguyen et al., 2017) for online learning of machine translation from weak user feedback to predicted translations, instead of from costly manually created reference translations.”
Counterfactual Learning from Bandit Feedback under Deterministic Logging : A Case Study in Statistical Machine Translation (2017, methodology): “learning protocol using the terminology of bandit structured prediction (Sokolov et al., 2016; Kreutzer et al., 2017), where at each round, a system (corresponding to a policy in RL terms) makes a prediction (also called action in RL, or pulling an arm of a bandit), and receives a reward, which is u”
A Reinforcement Learning Approach to Interactive-Predictive Neural Machine Translation (2018, methodology): “For the latter we rely on techniques from reinforcement learning (Sutton and Barto, 2017), a.k.a. bandit structured prediction (Sokolov et al., 2016; Kreutzer et al., 2017; Nguyen et al., 2017) in the context of sequence-to-sequence learning.”
A Shared Task on Bandit Learning for Machine Translation (2017, methodology): “The goal of this shared task is to evaluate existing algorithms for learning MT systems from weak feedback (Sokolov et al., 2015, 2016a; Kreutzer et al., 2017) on real-world data and compare them to new algorithms, with a focus on performing online learning efﬁciently and effectively from bandit…”
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning (2018, methodology): “Preference-based SPPI has been applied to multiple NLP applications, including text classiﬁcation, chunking and machine translation (Sokolov et al., 2016a; Kreutzer et al., 2017).”
Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translation (2021, methodology): “It has furthermore been identi-ﬁed as a promising tool to adapt pretrained models to new domains or user preferences by replacing reward functions with human feedback in human-in-the-loop learning (Sokolov et al., 2016; Nguyen et al., 2017).”