Distributional Reinforcement Learning for Energy-Based Sequential Models
By Tetiana Parshakova, Jean-Marc Andreoli, Marc Dymetman. 2019
23 citations recorded by the source corpus.
Global Autoregressive Models (GAMs) are a recent proposal [Parshakova et al., CoNLL 2019] for exploiting global properties of sequences for data-efficient learning of seq2seq models. In the first phase of training, an Energy-Based model (EBM) over sequences is derived. This EBM has high representational power, but is unnormalized and cannot be directly exploited for sampling. To address this issue [Parshakova et al., CoNLL 2019] proposes a distillation technique, which can only be applied under limited conditions. By relating this problem to Policy Gradient techniques in RL, but in a \emph{distributional} rather than \emph{optimization} perspective, we propose a general approach applicable to any sequential EBM. Its effectiveness is illustrated on GAM-based experiments.
Subfields: Artificial Intelligence, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Tomasz Korbak
- Germán Kruszewski
- Yilun Du
- Igor Mordatch
- Marc Dymetman
- Hady ElSahar
- Bryan Eikema
- Shuang Li
In Reinforcement Learning
- Germán Kruszewski
- Yilun Du
- Bryan Eikema
- Tomasz Korbak
- Hady ElSahar
- Marc Dymetman
- Gido M. van de Ven
- Jos Rozen
Citations in the corpus
- Aligning Language Models with Preferences through f-divergence Minimization (2023, methodology): “8 under the name f -DPG, in reference to the original DPG (Distributional Policy Gradient) approach introduced in (Parshakova et al., 2019), which can be seen as a special case of f -DPG (“KL-DPG”) with Df (πθ||p) set to KL(p||πθ) as discussed in Sec.”
- Controlling Conditional Language Models without Catastrophic Forgetting (2021, methodology): “This ablation resembles the original DPG algorithm for unconditional EBMs developed by (Parshakova et al., 2019a) and extended by (Khalifa et al., 2021).”
- Energy-Based Models for Code Generation under Compilability Constraints (2021, methodology): “Parshakova et al. (2019b) and Deng et al. (2020) augment a autoregressive language models with an additional global factor to obtain a lower perplexity on the training data.”
- Energy-Based Models for Continual Learning (2020, background): “…broadly speaking, EBMs can go out of the scope of classification problems, and similar ideas can be applied to regression (Gustafsson et al., 2020), generation (Xie et al., 2016; Du et al., 2020), and reinforcement learning (Parshakova et al., 2019), although we do not explore them in this work.”
- An approximate sampler for energy-based models with divergence diagnostics (2022, methodology): “The vector λ is found using self-normalized importance sampling (Owen, 2013; Parshakova et al., 2019a) and stochastic optimization.”