TravelPlanner: A Benchmark for Real-World Planning with Language Agents
By Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, Yu Su. International Conference on Machine Learning, 2024
433 citations recorded by the source corpus.
Planning has been part of the core pursuit for artificial intelligence since its conception, but earlier AI agents mostly focused on constrained settings because many of the cognitive substrates necessary for human-level planning have been lacking. Recently, language agents powered by large language models (LLMs) have shown interesting capabilities such as tool use and reasoning. Are these language agents capable of planning in more complex settings that are out of the reach of prior AI agents? To advance this investigation, we propose TravelPlanner, a new planning benchmark that focuses on travel planning, a common real-world planning scenario. It provides a rich sandbox environment, various tools for accessing nearly four million data records, and 1,225 meticulously curated planning intents and reference plans. Comprehensive evaluations show that the current language agents are not yet capable of handling such complex planning tasks-even GPT-4 only achieves a success rate of 0.6%. Language agents struggle to stay on task, use the right tools to collect information, or keep track of multiple constraints. However, we note that the mere possibility for language agents to tackle such a complex problem is in itself non-trivial progress. TravelPlanner provides a challenging yet meaningful testbed for future language agents.
Subfields: Artificial Intelligence
View Paper: https://www.semanticscholar.org/paper/11155af5ccd1889277f4269f6bb349a7633554f4
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
Citations in the corpus
- Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents (2024, positive): “This approach is widely adopted in benchmarks (Yang et al., 2024d; Patil et al., 2023; Tang et al., 2023; Basu et al., 2024; Xie et al., 2024a).”
- To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning (2024, neutral): “…I.1 L ONG H ORIZON P LANNING One set of tasks where symbolic reasoning helps substantially that our experiments haven’t covered as thoroughly (with the exception of BiGGen-Bench) is long-horizon planning (Valmeekam et al., 2023; Xie et al., 2024; Gundawar et al., 2024; Valmeekam et al., 2024).”
- TrajPrism: A Multi-Task Benchmark for Language-Grounded Urban Trajectory Understanding (2026, unknown): “TravelPlanner [28] evaluates LLM agents on multi-day itinerary planning with tool use, Semantic Routing [32] pairs synthetic natural-language queries with route generation, and MobilityBench [22] tests route-planning agents on real user queries.”