OpenAGI: When LLM Meets Domain Experts

By Yingqiang Ge, Wenyue Hua, Kai Mei, Jianchao Ji, Juntao Tan, Shuyuan Xu, Zelong Li, Yongfeng Zhang. Neural Information Processing Systems, 2023

353 citations recorded by the source corpus.

Human intelligence excels at combining basic skills to solve complex tasks. This capability is vital for Artificial Intelligence (AI) and should be embedded in comprehensive intelligent models, enabling them to harness expert models for complex task-solving towards Artificial General Intelligence (AGI). Large Language Models (LLMs) show promising learning and reasoning abilities, and can effectively use external models, tools or APIs to tackle complex problems. In this work, we introduce OpenAGI, an open-source AGI research platform designed for multi-step, real-world tasks. Specifically, OpenAGI uses a dual strategy, integrating standard benchmark tasks for benchmarking and evaluation, and open-ended tasks including more expandable models, tools or APIs for creative problem-solving. Tasks are presented as natural language queries to the LLM, which then selects and executes appropriate models. We also propose a Reinforcement Learning from Task Feedback (RLTF) mechanism that uses task results to improve the LLM's ability, which creates a self-improving AI feedback loop. While we acknowledge that AGI is a broad and multifaceted research challenge with no singularly defined solution path, the integration of LLMs with domain-specific expert models, inspired by mirroring the blend of general and specialized intelligence in humans, offers a promising approach towards AGI. We are open-sourcing the OpenAGI project's code, dataset, benchmarks, evaluation methods, and demo to foster community involvement in AGI advancement: https://github.com/agiresearch/OpenAGI.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/38179848e2d6a3ad373b1793848816111428ac36

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Peng Gao
2. Hongsheng Li
3. Yu Qiao
4. Renrui Zhang
5. Jiaming Han
6. Pan Lu
7. Shijie Geng
8. Yu Su

Citations in the corpus

• A Comprehensive Overview of Large Language Models (2023, neutral): “action [176], [177], [178], [179], task planning [180], [181], [182], navigation [183], [184], and learning [185], [186].”
• TravelPlanner: A Benchmark for Real-World Planning with Language Agents (2024, methodology): “…2015; Cobbe et al., 2021; Patel et al., 2021); tool-use evaluating agents’ proficiency in employing tools and reporting results (Li et al., 2023; Xu et al., 2023; Zhuang et al., 2023); and web navigation, testing agents’ ability to locate specific websites (Deng et al., 2023; Zhou et al., 2024;…”
• LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model (2023, background): “In recent years, with the continual expansion of LLMs’ capabilities, there has been a growing trend [14, 18, 21, 27, 37, 42, 51, 58, 61, 67, 69, 79] to combine them with visual foundation models and leverage their combined strengths to tackle more complex visionlanguage tasks.”
• Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents (2024, positive): “This has paved the way for developing AI agents that combine LLMs with tools and memory mechanisms capable of interacting with broader environments (Ge et al., 2023a; Xu et al., 2025a).”
• MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI (2023, background): “Rapid advances in large language models (LLMs) [13, 54, 67] have sparked broad discussions on the controversial concept of artificial general intelligence (AGI), often used to describe AI systems that perform on par or surpass humans at most tasks [1, 7, 21, 29, 49, 52].”
• The rise and potential of large language model based agents: a survey (2025, neutral): “For example, search-based tools can improve the scope and quality of the knowledge accessible to the agents with the aid of external databases, knowledge graphs, and web pages, while domain-specific tools can enhance an agent’s expertise in the corresponding field [243,244].”
• Mind2Web: Towards a Generalist Agent for the Web (2023, methodology): “These advanced tools could be subsequently employed by another LLM for more challenging problem-solving tasks [11, 30].”
• A survey on large language model based autonomous agents (2023, neutral): “OpenAGI [51] proposes a reinforcement learning with task feedback (RLTF) mechanism that utilizes manual or benchmark evaluation to improve the capabilities of the LLM-based agent.”
• Language is All a Graph Needs (2023, background): “In recent years, the AI community has witnessed the emergence of numerous powerful pre-trained Large Language Models (LLMs) (Devlin et al., 2018; Raffel et al., 2020; Brown et al., 2020; Tou-vron et al., 2023; Ouyang et al., 2022), which are driving huge advancements and lead to the pursuit of Artif”