Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learning
By Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, Yue Zhang. 2020
410 citations recorded by the source corpus.
A compelling approach to complex question answering is to convert the question to a sequence of actions, which can then be executed on the knowledge base to yield the answer, aka the programmer-interpreter approach. Use similar training questions to the test question, meta-learning enables the programmer to adapt to unseen questions to tackle potential distributional biases quickly. However, this comes at the cost of manually labeling similar questions to learn a retrieval model, which is tedious and expensive. In this paper, we present a novel method that automatically learns a retrieval model alternately with the programmer from weak supervision, i.e., the system’s performance with respect to the produced answers. To the best of our knowledge, this is the first attempt to train the retrieval model with the programmer jointly. Our system leads to state-of-the-art performance on a large-scale task for complex question answering over knowledge bases. We have released our code at https://github.com/DevinJake/MARL.
Subfields: Artificial Intelligence, Molecular Biology, Retrieval-Augmented Generation, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Percy Liang
- Jianfeng Gao
- Yejin Choi
- Hannaneh Hajishirzi
- Noah A. Smith
- Graham Neubig
In Molecular Biology
In Retrieval-Augmented Generation
- Tushar Khot
- Luke S. Zettlemoyer
- Yizhong Wang
- Niklas Muennighoff
- Weizhu Chen
- Duyu Tang
- Lukasz Kaiser
- Matt Gardner
In Natural Language Processing
- Mohit Bansal
- Matt Gardner
- Hannaneh Hajishirzi
- Jonathan Berant
- Luke S. Zettlemoyer
- Tushar Khot
- Ashish Sabharwal
- Percy Liang
Citations in the corpus
- Reasoning Like Program Executors (2022, background): “LogiQA A multi-choice reading comprehension dataset that evaluates the logical reasoning ability, whose questions are designed by domain experts (Liu et al., 2020).”
- Training Verifiers to Solve Math Word Problems (2021, background): “Other recent reasoning-related datasets have focused on mathematical reasoning on symbolic math (Lample and Charton, 2019), reading comprehension (LogiQA) (Liu et al., 2020), and commonsense question answering (Common-senseQA) (Talmor et al., 2018).”
- GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, neutral): “Due to space constraints a representative subset of the results are shown here, with the rest in Appendix E. Natural Language Tasks We evaluate our model on a diverse collection of standard language model evaluation datasets: ANLI (Nie et al., 2020), ARC (Clark et al., 2018), HeadQA (English) (Vilar”
- Measuring Coding Challenge Competence With APPS (2021, background): “Liu et al. (2020a) do a smaller-scale analysis of code generation but with their limited language-specific training data models “fail to pass even a single predefined test case” on their 300 test problems, while with our large training set and test set, trained models can pass tens of thousands of…”
- Reasoning with Transformer-based Models: Deep Learning, but Shallow Reasoning (2021, background): “[Liu et al., 2020b].”
- Logic-Driven Context Extension and Data Augmentation for Logical Reasoning of Text (2021, methodology): “LogiQA (Liu et al., 2020) is collected from the National Civil Servants Examination of China covering 5 logical reasoning types.”
- Analytical Reasoning of Text (2022, background): “LogiQA105 (Liu et al., 2020b) and ReClor (Yu et al., 2020) are 106 sourced from examination in realistic scenario and 107 examine a range of logical reasoning skills.”
- OLMo: Accelerating the Science of Language Models (2024, background): “These tasks are headqa_en (Vilares and Gómez-Rodríguez, 2019), logiqa (Liu et al., 2020), mrpc (Dolan and Brock-ett, 2005), qnli (Wang et al., 2018), wic (Pilehvar and Camacho-Collados, 2018), and wnli (Wang et al., 2018).”
- Many-shot Jailbreaking (2024, background): “We observe that in-context learning follows predictable power laws on the following safety-unrelated datasets: LogiQA (Liu et al., 2020), TruthfulQA (Lin et al., 2022), Winogrande (Sakaguchi et al., 2019), TriviaQA (Joshi et al., 2017), Common-senseQA (Talmor et al., 2019), CREAK (Onoe et al.,…”
- On the Paradox of Learning to Reason from Data (2022, methodology): “Prior work contextualizes the problem of logical reasoning by proposing reasoning-dependent datasets and studies solving the tasks with neural models (Johnson et al., 2017; Sinha et al., 2019; Yu et al., 2020; Liu et al., 2020; Tian et al., 2021).”
- Reasoning with Latent Structure Refinement for Document-Level Relation Extraction (2020, methodology): “…entities within a single sentence by modeling interactions in the input sequence (Zeng et al., 2014; Wang et al., 2016; Zhou et al., 2016; Zhang et al., 2017; Guo et al., 2020) or the corresponding dependency tree (Xu et al., 2015a,b; Liu et al., 2015; Miwa and Bansal, 2016; Zhang et al., 2018).”
- Semi-supervised Relation Extraction via Incremental Meta Self-Training (2020, neutral): “Relation extraction methods are often formulated in a supervised setting (Zeng et al., 2015; Guo et al., 2020; Nan et al., 2020) and require manual annotations on large amounts of corpora, which is labor-intensive to obtain.”