Cold-Start Reinforcement Learning with Softmax Policy Gradient

By Nan Ding, Radu Soricut. 2017

47 citations recorded by the source corpus.

Policy-gradient approaches to reinforcement learning have two common and undesirable overhead procedures, namely warm-start training and sample variance reduction. In this paper, we describe a reinforcement learning method based on a softmax value function that requires neither of these procedures. Our method combines the advantages of policy-gradient methods with the efficiency and simplicity of maximum-likelihood approaches. We apply this new cold-start reinforcement learning method in training sequence generation models for structured output prediction problems. Empirical evidence validates this method on automatic summarization and image captioning tasks.

Subfields: Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning (2018, methodology): “A large number of DNN models for image caption generation have been trained and evaluated using COCO captions (Vinyals et al., 2015a; Fang et al., 2015; Xu et al., 2015; Ranzato et al., 2015; Yang et al., 2016; Liu et al., 2017; Ding and Soricut, 2017).”
Optimal Completion Distillation for Sequence Learning (2018, background): “SPG (Ding and Soricut, 2017) changes the policy gradient formulation to sample from a reward shaped model distribution.”
Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsing (2018, background): “Standard solutions to the small gradient problem involves supervised pretraining [52, 19, 46] or using supervised data to generate rewarding samples [36, 13], which cannot be applied when supervised data are not available.”
Text Generation with Efficient (Soft) Q-Learning (2021, background): “Cold-start [10] proposed softmax policy gradient (SPG) that does not rely on MLE pretraining but requires various dedicated techniques for effective training (e.g., token-level decomposition of sequence reward, etc).”
Generalization in Generation: A closer look at Exposure Bias (2019, background): “Shortcomings of maximum-likelihood training for sequence generation have often been discussed (Ding and Soricut, 2017; Leblond et al., 2018; Ranzato et al., 2016), but without pointing to generalization as the key aspect.”