PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models
By Torsten Scholak, Nathan Schucher, Dzmitry Bahdanau. 2021
624 citations recorded by the source corpus.
Large pre-trained language models for textual data have an unconstrained output space; at each decoding step, they can produce any of 10,000s of sub-word tokens. When fine-tuned to target constrained formal languages like SQL, these models often generate invalid code, rendering it unusable. We propose PICARD (code available at https://github.com/ElementAI/picard), a method for constraining auto-regressive decoders of language models through incremental parsing. PICARD helps to find valid output sequences by rejecting inadmissible tokens at each decoding step. On the challenging Spider and CoSQL text-to-SQL translation tasks, we show that PICARD transforms fine-tuned T5 models with passable performance into state-of-the-art solutions.
Subfields: Artificial Intelligence, Natural Language Processing, Information Extraction
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Luke S. Zettlemoyer
- Percy Liang
- Graham Neubig
- Mohit Bansal
- Caiming Xiong
- Jianfeng Gao
- Richard Socher
- Noah A. Smith
In Natural Language Processing
- Luke S. Zettlemoyer
- Percy Liang
- Graham Neubig
- Caiming Xiong
- Wen-tau Yih
- Richard Socher
- Jonathan Berant
- Mohit Bansal
In Information Extraction
- Sida I. Wang
- Wen-tau Yih
- Luke S. Zettlemoyer
- Nitarshan Rajkumar
- Raymond Li
- Jonathan Herzig
- Dzmitry Bahdanau
- Panupong Pasupat
Citations in the corpus
- UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022, methodology): “This phenomenon is also observed by Scholak et al. (2021), who used constrained decoding to improve the validity, largely improving the parsing performance.”
- Teaching Large Language Models to Self-Debug (2023, methodology): “While models designed and trained for specialized domains have achieved impressive performance in various applications such as text-to-code generation (Li et al., 2023a; Wang et al., 2020; Scholak et al., 2021; Dong & Lapata, 2016; Iyer et al., 2018) and code translation (Chen et al.”
- Evaluating the Text-to-SQL Capabilities of Large Language Models (2022, methodology): “The 3B parameter T5 model is currently the state-of-the-art on Spider when combined with constrained inference using the PICARD algorithm (Scholak et al., 2021).”
- DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction (2023, neutral): “Despite a strong performance, LLMs fall behind, compared to existing methods (Scholak et al., 2021; Li et al., 2023a), especially on medium and complex queries.”
- Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs (2023, methodology): “We concatenate the question with serialized database schema as input [41, 51, 42].”
- RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL (2023, methodology): “T5-3B (Scholak, Schucher, and Bahdanau 2021) 71.5 74.4 68.0 70.1 T5-3B + PICARD (Scholak, Schucher, and Bahdanau 2021) 75.5 79.3 71.9 75.1 RASAT + PICARD (Qi et al. 2022) 75.3 80.5 70.9 75.5 Our proposed method ).”
- CodeS: Towards Building Open-source Language Models for Text-to-SQL (2024, neutral): “…users who may not be familiar with SQL or the structure of a database to interact with the database using natural language, and thus it has garnered increasing attention from both the database [4, 18, 24] and natural language processing communities [36, 57]. as data must be sent to model providers.”
- Binding Language Models in Symbolic Languages (2022, methodology): “…grounding tasks mainly adopt semantic parsing since symbolic languages like SQL, SPARQL can be executed on them (Berant et al., 2013; Liang et al., 2017; Yin & Neubig, 2017; Zhong et al., 2018; Yu et al., 2018; Shaw et al., 2021; Scholak et al., 2021 (Rajkumar et al., 2022; Shi et al., 2022).”
- LEVER: Learning to Verify Language-to-Code Generation with Execution (2023, positive): “Previous Work without Finetuning Rajkumar et al. (2022) 67.0 MBR-Exec (Shi et al., 2022) 75.2 Coder-Reviewer (Zhang et al., 2022) 74.5 Previous Work with Finetuning T5-3B (Xie et al., 2022) 71.8 PICARD (Scholak et al., 2021) 75.5 RASAT (Qi et al., 2022) 80.5 This Work with code-davinci-002 Greedy 75”
- SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL (2024, methodology): “Fine-tuning baselines : PICARD [3] employs incremental parsing to constrain auto-regressive decoding.”
- Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation (2023, methodology): “With rapid advancement of deep learning, numerous techniques are applied to help Text-to-SQL task, such as attention mechanism [23], graph representation [14, 20, 32, 42, 46, 50], syntax parsing [12, 19, 35, 43], etc.”
- Natural Language to Code Translation with Execution (2022, methodology): “…(2021), where they use all 374 training examples to finetune their pretrained code model; for Spider, we compare to the current state-of-the-art result (Scholak et al., 2021); for NL2Bash, we finetune GPT-2 (Radford et al., 2019) with all training examples with the same prompting set up as Table 1.”
- Synchromesh: Reliable code generation from pre-trained language models (2022, background): “In concurrent work, Scholak et al. (2021) applied similar semantic constraints to synthesizing SQL queries.”
- Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing (2022, methodology): “…of predictions are still syntactically incorrect, especially on SMCalFlow-CS, suggesting using constrained decoding to prevent generating invalid outputs can be an effective solu-tion to improve performance in semantic parsing tasks (Shin et al., 2021; Shin and Durme, 2021; Scholak et al., 2021).”