ToolQA: A Dataset for LLM Question Answering with External Tools
By Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, Chao Zhang. Neural Information Processing Systems, 2023
396 citations recorded by the source corpus.
Large Language Models (LLMs) have demonstrated impressive performance in various NLP tasks, but they still suffer from challenges such as hallucination and weak numerical reasoning. To overcome these challenges, external tools can be used to enhance LLMs' question-answering abilities. However, current evaluation methods do not distinguish between questions that can be answered using LLMs' internal knowledge and those that require external information through tool use. To address this issue, we introduce a new dataset called ToolQA, which is designed to faithfully evaluate LLMs' ability to use external tools for question answering. Our development of ToolQA involved a scalable, automated process for dataset curation, along with 13 specialized tools designed for interaction with external knowledge in order to answer questions. Importantly, we strive to minimize the overlap between our benchmark data and LLMs' pre-training data, enabling a more precise evaluation of LLMs' tool-use reasoning abilities. We conducted an in-depth diagnosis of existing tool-use LLMs to highlight their strengths, weaknesses, and potential improvements. Our findings set a new benchmark for evaluating LLMs and suggest new directions for future advancements. Our data and code are freely available to the broader scientific community on GitHub.
Subfields: Artificial Intelligence, Information Systems
View Paper: https://www.semanticscholar.org/paper/c12b80b44d9acfe6cd92fdf965264c4b706c367c
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Information Systems
Citations in the corpus
- Large Language Models: A Survey (2024, neutral): “• ToolQA [199] is a question answering benchmark to evaluate LLMs’ ability to use external tools for answering questions.”
- Knowledge Conflicts for LLMs: A Survey (2024, positive): “Mean-while, LLMs continue to engage with external contextual knowledge after deployed (Pan et al., 2022), including user prompts (Liu et al., 2023a), documents from the Web (Shi et al., 2023c), or tools (Schick et al., 2023; Zhuang et al., 2023).”
- GAIA: a benchmark for General AI Assistants (2023, background): “ToolQA (Zhuang et al., 2023) or Gentopia (Xu et al., 2023a) for example combine existing datasets with human annotations (MMLU, MATH, etc.) at the risk of contamination during training, and without ensuring tool usage is actually tested.”
- OpenHands: An Open Platform for AI Software Developers as Generalist Agents (2024, neutral): “ToolQA (Zhuang et al., 2024) evaluates agents’ abilities to use external tools.”
- TravelPlanner: A Benchmark for Real-World Planning with Language Agents (2024, methodology): “In the two-stage mode, we use the ReAct (Yao et al., 2022) framework for information collection, which is recognized for its effective iteration with tools (Zhuang et al., 2023) while varying the foundation LLMs.”
- Tool learning with large language models: a survey (2024, neutral): “Conversely, other studies treat each API as an independent tool [19, 30, 122].”
- The application of large language models in medicine: A scoping review (2024, neutral): “49 Their ability to automatically score objective questions, such as multiple-choice or true-or-false formats, and provide preliminary evaluations for short-answer and essay responses, underscores their utility in educational settings.”
- Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, background): “In the field of NLP, Large Language Models (LLMs) have achieved significant advancements across various tasks including text generation [10], [243], translation [239], [61], personalized chat-bots [192], [103], [187], and summarization [212], demonstrating remarkable proficiency.”