Energy-Based Models for Code Generation under Compilability Constraints

By Tomasz Korbak, Hady ElSahar, Marc Dymetman, Germán Kruszewski. 2021

14 citations recorded by the source corpus.

Neural language models can be successfully trained on source code, leading to applications such as code completion. However, their versatile autoregressive self-supervision objective overlooks important global sequence-level features that are present in the data such as syntactic correctness or compilability. In this work, we pose the problem of learning to generate compilable code as constraint satisfaction. We define an Energy-Based Model (EBM) representing a pre-trained generative model with an imposed constraint of generating only compilable sequences. We then use the KL-Adaptive Distributional Policy Gradient algorithm (Khalifa et al., 2021) to train a generative model approximating the EBM. We conduct experiments showing that our proposed approach is able to improve compilability rates without sacrificing diversity and complexity of the generated samples.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tomasz Korbak
2. Germán Kruszewski
3. Hady ElSahar
4. Marc Dymetman
5. Ethan Perez
6. C. L. Buckley
7. Bryan Eikema
8. Yonatan Belinkov

In Information Systems

1. Tomasz Korbak
2. Germán Kruszewski
3. Marc Dymetman
4. Hady ElSahar
5. Ethan Perez

In Natural Language Processing

1. Tomasz Korbak
2. Germán Kruszewski
3. Marc Dymetman
4. Hady ElSahar
5. Ethan Perez
6. C. L. Buckley
7. Yonatan Belinkov
8. Bryan Eikema

In Generative Models

1. Tomasz Korbak
2. Germán Kruszewski
3. Hady ElSahar
4. Marc Dymetman
5. Ethan Perez
6. C. L. Buckley

Citations in the corpus

• On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting (2022, methodology): “Then the forward KL divergence is minimized between this target distribution and an auto-regressive policy using a family of algorithms referred to as Distributional Policy Gradients (DPG) (Parshakova et al., 2019b; Khalifa et al., 2021; Korbak et al., 2021, 2022a).”
• RL with KL penalties is better viewed as Bayesian inference (2022, background): “, 2019), to code generation (Korbak et al., 2021) and LM debiasing (Khalifa et al.”
• Controlling Conditional Language Models without Catastrophic Forgetting (2021, methodology): “Most in line with our work, Korbak et al. (2021) used DPG to improve compilability of unconditional language models for code.”