Binding Language Models in Symbolic Languages

By Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, R. Nadkarni, Yushi Hu, Caiming Xiong, Dragomir R. Radev, Mari Ostendorf, Luke S. Zettlemoyer and 1 more. 2022

322 citations recorded by the source corpus.

Though end-to-end neural approaches have recently been dominating NLP tasks in both performance and ease-of-use, they lack interpretability and robustness. We propose Binder, a training-free neural-symbolic framework that maps the task input to a program, which (1) allows binding a unified API of language model (LM) functionalities to a programming language (e.g., SQL, Python) to extend its grammar coverage and thus tackle more diverse questions, (2) adopts an LM as both the program parser and the underlying model called by the API during execution, and (3) requires only a few in-context exemplar annotations. Specifically, we employ GPT-3 Codex as the LM. In the parsing stage, with only a few in-context exemplars, Codex is able to identify the part of the task input that cannot be answerable by the original programming language, correctly generate API calls to prompt Codex to solve the unanswerable part, and identify where to place the API calls while being compatible with the original grammar. In the execution stage, Codex can perform versatile functionalities (e.g., commonsense QA, information extraction) given proper prompts in the API calls. Binder achieves state-of-the-art results on WikiTableQuestions and TabFact datasets, with explicit output programs that benefit human debugging. Note that previous best systems are all finetuned on tens of thousands of task-specific samples, while Binder only uses dozens of annotations as in-context exemplars without any training. Our code is available at https://github.com/HKUNLP/Binder .

Subfields: Artificial Intelligence, Information Systems, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Large Language Models

In Natural Language Processing

Citations in the corpus

PAL: Program-aided Language Models (2022, methodology): “The closest work to ours technically may be Binder (Cheng et al., 2022), but it addressed mostly answering questions about tables using SQL and SQL-like Python.”
ViperGPT: Visual Inference via Python Execution for Reasoning (2023, methodology): “Using Codex as the program generator and generating code directly in Python allows us to draw on training at scale on the Internet, where Python code is abundant.”
Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks (2022, methodology): “Another concurrent work similar to ours is BINDER (Cheng et al., 2022), which applies Codex to synthesize ‘soft’ SQL queries to answer questions from tables.”
Large Language Models as Analogical Reasoners (2023, background): “…2023; Moor et al., 2023), retrieving structured knowledge (Lin et al., 2019; Feng et al., 2020; Zhang et al., 2022a; Yasunaga et al., 2021; 2022a; Xie et al., 2022), and incorporating external modules for reasoning such as logic and program execution (Chen et al., 2018; 2019; Yasunaga & Liang,…”
PromptCap: Prompt-Guided Task-Aware Image Captioning (2022, background): “This framework has attracted attention especially in few-shot natural language processing tasks [11, 20, 50].”
Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing (2022, methodology): “Tasks Prior works have evaluated on text-to-SQL semantic parsing tasks using LMs (Rajkumar et al., 2022; Cheng et al., 2022).”
L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models (2023, background): “Intuitively, semantic parsing challenges LLMs on grounded language understanding (Xie et al., 2022; Cheng et al., 2022), where a model needs to associate NL concepts in utterances (e.”
LEVER: Learning to Verify Language-to-Code Generation with Execution (2023, methodology): “Previous Work without Finetuning Codex QA∗ (Cheng et al., 2022) 50.”
Large Language Models are few(1)-shot Table Reasoners (2022, methodology): “One contemporary work similar to ours is BINDER (Cheng et al., 2022), which utilizes Codex to synthesize SQL to execute logical forms against tables for question answering.”
Natural Language to Code Generation in Interactive Data Science Notebooks (2022, background): “, 2022, inter alia), with promising applications in synthesizing code for data wrangling and EDA tasks (Jain et al., 2021; Rajkumar et al., 2022; Cheng et al., 2022).”
StructGPT: A General Framework for Large Language Model to Reason over Structured Data (2023, methodology): “For TableQA, we select MAPO (Liang et al., 2018), TAPAS (Herzig et al., 2020; Eisensch-los et al., 2020), UnifiedSKG (T5-3B) (Xie et al., 2022), TAPEX (Liu et al., 2022), and DATER (Ye et al., 2023).”
Parsel: A Unified Natural Language Framework for Algorithmic Reasoning (2022, background): “For example, Cheng et al. (2022) explored the introduction of a language-model-based evaluation function, which would allow f(’North America?’”
Modular Visual Question Answering via Code Generation (2023, background): “Cheng et al. (2023) introduces a framework for reasoning jointly over tables, text, and images, where the images are represented by image captions.”