Learning to Generalize from Sparse and Underspecified Rewards

By Rishabh Agarwal, Chen Liang, Dale Schuurmans, Mohammad Norouzi. 2019

105 citations recorded by the source corpus.

We consider the problem of learning from sparse and underspecified rewards, where an agent receives a complex input, such as a natural language instruction, and needs to generate a complex response, such as an action sequence, while only receiving binary success-failure feedback. Such success-failure rewards are often underspecified: they do not distinguish between purposeful and accidental success. Generalization from underspecified rewards hinges on discounting spurious trajectories that attain accidental success, while learning from sparse feedback requires effective exploration. We address exploration by using a mode covering direction of KL divergence to collect a diverse set of successful trajectories, followed by a mode seeking KL divergence to train a robust policy. We propose Meta Reward Learning (MeRL) to construct an auxiliary reward function that provides more refined feedback for learning. The parameters of the auxiliary reward function are optimized with respect to the validation performance of a trained policy. The MeRL approach outperforms our alternative reward learning technique based on Bayesian Optimization, and achieves the state-of-the-art on weakly-supervised semantic parsing. It improves previous work by 1.2% and 2.4% on WikiTableQuestions and WikiSQL datasets respectively.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

A Discrete Hard EM Approach for Weakly Supervised Question Answering (2019, background): “…an-6 (Liang et al., 2017) 70.1 Hard EM (Liang et al., 2018) 70.2 Beam-based MML (Liang et al., 2018) 70.7 MAPO (Liang et al., 2018) 71.8 72.4 MAPOX (Agarwal et al., 2019) 74.5 74.2 MAPOX+MeRL (Agarwal et al., 2019) 74.9 74.8 MML 70.6 70.5 Ours 84.4 83.9 Fully-supervised setting SQLNet (Xu et al.,…”
A Survey of Reinforcement Learning Informed by Natural Language (2019, background): “[Agarwal et al., 2019] perform a meta-optimisation to learn auxiliary rewards conditioned on features extracted from instructions.”
TabFact: A Large-scale Dataset for Table-based Fact Verification (2019, methodology): “They aim to retrieve answers by synthesizing programs and executing them on the tables, such as in Neural Programmer (Neelakan-tan et al., 2016; 2017) and Neural Symbolic Machines (Liang et al., 2017; 2018; Agarwal et al., 2019).”
GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing (2020, background): “…data understanding problems, such as table semantic parsing (Zhong et al., 2017; Yu et al., 2018b), question answering (Pasupat & Liang, 2015; Chen et al., 2020), retrieval (Zhang et al., 2019a), fact-checking (Chen et al., 2019) and summarization (Parikh et al., 2020; Radev et al., 2020).”
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data (2020, unknown): “3 Agarwal et al. (2019) 43.”
Neuroevolution of self-interpretable agents (2020, background): “[3, 44, 45, 51, 57, 69, 86, 101, 105]) that approach the generalization problem, and future work will continue to develop on these ideas to push the generalization abilities proposed in more general domains [7, 16, 17, 48, 58, 69, 104].”
Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models (2023, methodology): “IML has been shown to perform well with relatively small-scale language models for semantic parsing (Liang et al., 2016; Agarwal et al., 2019), machine translation (Wu et al., 2016) and simple math reasoning (Ni et al., 2022).”
Large Language Models are few(1)-shot Table Reasoners (2022, methodology): “Our results reveal that LLMs (Ouyang et al., 2022; Chen et al., 2021a; Chowdhery et al., 2022) can achieve striking performance with only 1 or 2 demonstrations, e.g. 48.8% on WikiTable-Questions and 78.8% on TabFact, which are on par some near-SoTA models (Yu et al., 2021; Eisen-schlos et al., 2020)”
Self-Attentional Credit Assignment for Transfer in Reinforcement Learning (2019, unknown): “Others learn auxiliary reward functions in the hope that they will enable transfer by imposing consistency in the reward (Houthooft et al., 2018; Hessel et al., 2018; Agarwal et al., 2019).”
Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning (2021, background): “In this context, various meta-learning approaches capable of zero-shot generalization have been proposed (Li et al., 2018; Agarwal et al., 2019; Balaji et al., 2018).”
Learning by Fixing: Solving Math Word Problems with Weak Supervision (2020, methodology): “Meta Reward Learning (Agarwal et al. 2019) uses an auxiliary reward function to provide feedback beyond a binary success or failure.”
TaPas: Weakly Supervised Table Parsing via Pre-training (2020, background): “…answering from semi-structured tables is usually seen as a semantic parsing task where the question is translated to a logical form that can be executed against the table to retrieve the correct denotation (Pasupat and Liang, 2015; Zhong et al., 2017; Dasigi et al., 2019; Agarwal et al., 2019).”