Learning a Natural Language Interface with Neural Programmer

By Arvind Neelakantan, Quoc V. Le, Martín Abadi, Andrew McCallum, Dario Amodei. 2016

139 citations recorded by the source corpus.

Learning a natural language interface for database tables is a challenging task that involves deep language understanding and multi-step reasoning. The task is often approached by mapping natural language queries to logical forms or programs that provide the desired response when executed on the database. To our knowledge, this paper presents the first weakly supervised, end-to-end neural network model to induce such programs on a real-world dataset. We enhance the objective function of Neural Programmer, a neural network with built-in discrete operations, and apply it on WikiTableQuestions, a natural language question-answering dataset. The model is trained end-to-end with weak supervision of question-answer pairs, and does not require domain-specific grammars, rules, or annotations that are key elements in previous approaches to program induction. The main experimental result in this paper is that a single Neural Programmer model achieves 34.2% accuracy using only 10,000 examples with weak supervision. An ensemble of 15 models, with a trivial combination technique, achieves 37.7% accuracy, which is competitive to the current state-of-the-art accuracy of 37.1% obtained by a traditional natural language semantic parser.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Programmable Agents (2017, background): “Compositional neural architectures paired with different forms of attention have led to impressive results in natural language interfaces for database tables, and language acquisition in 2D navigation environments [28, 37].”
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback (2018, methodology): “Semantic parsers have been successfully trained using neural sequence-to-sequence models with a cross-entropy objective and question-parse pairs (Jia and Liang, 2016; Dong and Lapata, 2016)) or question-answer pairs (Neelakantan et al., 2017).”
Deep Learning: A Critical Appraisal (2018, background): “…and recent trend towards systems such as differentiable neural computers (Graves et al., 2016), programming with differentiable interpreters (Bo š njak, Rocktäschel, Naradowsky, & Riedel, 2016), and neural programming with discrete operations (Neelakantan, Le, Abadi, McCallum, & Amodei, 2016).”
Search-based Neural Structured Learning for Sequential Question Answering (2017, background): “Neural programmer: The neural programmer proposed by Neelakantan et al. (2017) has shown promising results on WTQ, achieving accuracies on par with those of FP .”
Neural Module Networks for Reasoning over Text (2019, methodology): “These have also been extended for QA using symbolic reasoning against semi-structured tables (Pasupat & Liang, 2015; Krishnamurthy et al., 2017; Neelakantan et al., 2016).”
Complex Sequential Question Answering: Towards Learning to Converse Over Linked Question Answer Pairs with a Knowledge Graph (2018, background): “Recently (Neelakantan et al. 2016) have explored complex question answering over around 18.5K queries from the Wik-iTableQuestions dataset, but their tables have less than 100 rows and a handful of columns whereas our complex QAs are grounded in a KB of over 20 million tuples.”
Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models (2021, background): “However, much of this prior work is limited to pre-determined composition structures (Berant et al., 2013; Seo et al., 2015; Neelakantan et al., 2017; Roy and Roth, 2018).”
Neural Program Synthesis with Priority Queue Training (2018, background): “…there have been promising results on the task of binary search (Nachum et al., 2017a), sorting an array of numbers (Reed & de Freitas, 2016; Cai et al., 2017), solving simple Q&A from tables (Neelakantan et al., 2016; 2017), visual Q&A (Andreas et al., 2016; Hu et al., 2017; Johnson et al.)”
Did the Model Understand the Question? (2018, methodology): “We analyze a model called Neural Program-mer (NP) (Neelakantan et al., 2017).”
Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsing (2018, methodology): “For the Lisp interpreter, we added functions according to [67, 34] for W IKI T ABLE Q UESTIONS experiments and used the subset of functions equivalent to column selection, aggregation, and conditions for W IKI SQL.”
PlotQA: Reasoning over Scientific Plots (2019, background): “There are recent neural approaches for answering questions over semi-structured tables such as [24, 8].”
Program Synthesis from Natural Language Using Recurrent Neural Networks (2017, background): “However, it has recently been shown that deeplearning based approaches work equally well, for example to produce regular expressions [16] and database queries [6, 23].”
LEGO: Latent Execution-Guided Reasoning for Multi-Hop Question Answering on Knowledge Graphs (2021, background): “…have been proposed to improve the model performance with weak supervision, i.e., the training supervision only contains the ground truth execution results on tabular databases (Liang et al., 2017; Neelakantan et al., 2017; Krishnamurthy et al., 2017; Guu et al., 2017; Liang et al., 2018).”
Learning Semantic Parsers from Denotations with Latent Structured Alignments and Abstract Programs (2019, unknown): “Following previous work (Neelakantan et al., 2017; Liang et al., 2018), we also add an indicator feature specifying whether the word appears in the table.”
Published as a conference paper at ICLR 2018 S IMULATING A CTION D YNAMICS WITH N EURAL P ROCESS N ETWORKS (2018, methodology): “Neural Programmers (Neelakantan et al., 2015; 2016) have also used functions to simulate reasoning, by building a model to select rows in a database and applying operation on those selected rows.”
A Comprehensive Evaluation of Tool-Assisted Generation Strategies (2023, unknown): “There is adjacent research on augmenting neural networks, in ways besides textual interfaces, with tools (e.g., Andor et al., 2019; Jacovi et al., 2019) or training differentiable subnetworks that heavily mimic tools (Neelakantan et al., 2017; Trask et al., 2018).”