Learning Structured Predictors from Bandit Feedback for Interactive NLP

By Artem Sokolov, Julia Kreutzer, Christopher Lo, Stefan Riezler. 2016

34 citations recorded by the source corpus.

Structured prediction from bandit feed-back describes a learning scenario where instead of having access to a gold standard structure, a learner only receives partial feedback in form of the loss value of a predicted structure. We present new learning objectives and algorithms for this interactive scenario, focusing on convergence speed and ease of elicitability of feed-back. We present supervised-to-bandit simulation experiments for several NLP tasks (machine translation, sequence labeling, text classiﬁcation), showing that bandit learning from relative preferences eases feedback strength and yields improved empirical convergence.

Subfields: Artificial Intelligence, Management Science and Operations Research, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Can Neural Machine Translation be Improved with User Feedback? (2018, methodology): “…signal for response prediction in online advertising (Bottou et al., 2013), bandit learning for machine translation has so far been restricted to simulation experiments (Sokolov et al., 2016b; Lawrence et al., 2017b; ∗The work for this paper was done while the first author was an intern at eBay.”
Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning (2018, background): “Our aim is to show that successful learning from simulated bandit feedback (Sokolov et al., 2016b; Kreutzer et al., 2017; Nguyen et al., 2017; Lawrence et al., 2017) does in fact carry over to learning from actual human bandit feedback.”
Bandit Structured Prediction for Neural Sequence-to-Sequence Learning (2017, methodology): “In the following, we will present objectives from Sokolov et al. (2016b) transferred to neural models, and explain how they can be enhanced by con-trol variates.”
Stochastic Structured Prediction under Bandit Feedback (2016, methodology): “This idea is formalized in [19] as an expected loss objective with respect to a conditional distribution of pairs of structured outputs.”
A Survey of Preference-Based Reinforcement Learning Methods (2017, methodology): “, 2010; Shamir, 2017), which has recently been extended to structured prediction problems (Sokolov et al., 2016).”
A Shared Task on Bandit Learning for Machine Translation (2017, methodology): “The goal of this shared task is to evaluate existing algorithms for learning MT systems from weak feedback (Sokolov et al., 2015, 2016a; Kreutzer et al., 2017) on real-world data and compare them to new algorithms, with a focus on performing online learning efﬁciently and effectively from bandit…”
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning (2018, methodology): “Preference-based SPPI has been applied to multiple NLP applications, including text classiﬁcation, chunking and machine translation (Sokolov et al., 2016a; Kreutzer et al., 2017).”
Survey on reinforcement learning for language processing (2021, background): “In the case of the training, when it is done interactively, there is evidence that reinforcement learning can be used to improve the real-time translation performance after several interactions with humans [36, 111, 110].”
Preference-based interactive multi-document summarisation (2019, methodology): “1 shows this general idea in comparison to the state-of-the-art preference-based interactive NLP paradigm, Structured Prediction from Partial Information (SPPI) (Sokolov et al. 2016b; Kreutzer et al. 2017).”