PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models

By Torsten Scholak, Nathan Schucher, Dzmitry Bahdanau. 2021

624 citations recorded by the source corpus.

Large pre-trained language models for textual data have an unconstrained output space; at each decoding step, they can produce any of 10,000s of sub-word tokens. When fine-tuned to target constrained formal languages like SQL, these models often generate invalid code, rendering it unusable. We propose PICARD (code available at https://github.com/ElementAI/picard), a method for constraining auto-regressive decoders of language models through incremental parsing. PICARD helps to find valid output sequences by rejecting inadmissible tokens at each decoding step. On the challenging Spider and CoSQL text-to-SQL translation tasks, we show that PICARD transforms fine-tuned T5 models with passable performance into state-of-the-art solutions.

Subfields: Artificial Intelligence, Natural Language Processing, Information Extraction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Luke S. Zettlemoyer
  2. Percy Liang
  3. Graham Neubig
  4. Mohit Bansal
  5. Caiming Xiong
  6. Jianfeng Gao
  7. Richard Socher
  8. Noah A. Smith

In Natural Language Processing

  1. Luke S. Zettlemoyer
  2. Percy Liang
  3. Graham Neubig
  4. Caiming Xiong
  5. Wen-tau Yih
  6. Richard Socher
  7. Jonathan Berant
  8. Mohit Bansal

In Information Extraction

  1. Sida I. Wang
  2. Wen-tau Yih
  3. Luke S. Zettlemoyer
  4. Nitarshan Rajkumar
  5. Raymond Li
  6. Jonathan Herzig
  7. Dzmitry Bahdanau
  8. Panupong Pasupat

Citations in the corpus