LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory

By Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, Dong Yu. International Conference on Learning Representations, 2024

429 citations recorded by the source corpus.

Recent large language model (LLM)-driven chat assistant systems have integrated memory components to track user-assistant chat histories, enabling more accurate and personalized responses. However, their long-term memory capabilities in sustained interactions remain underexplored. We introduce LongMemEval, a comprehensive benchmark designed to evaluate five core long-term memory abilities of chat assistants: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. With 500 meticulously curated questions embedded within freely scalable user-assistant chat histories, LongMemEval presents a significant challenge to existing long-term memory systems, with commercial chat assistants and long-context LLMs showing a 30% accuracy drop on memorizing information across sustained interactions. We then present a unified framework that breaks down the long-term memory design into three stages: indexing, retrieval, and reading. Built upon key experimental insights, we propose several memory design optimizations including session decomposition for value granularity, fact-augmented key expansion for indexing, and time-aware query expansion for refining the search scope. Extensive experiments show that these optimizations greatly improve both memory recall and downstream question answering on LongMemEval. Overall, our study provides valuable resources and guidance for advancing the long-term memory capabilities of LLM-based chat assistants, paving the way toward more personalized and reliable conversational AI. Our benchmark and code are publicly available at https://github.com/xiaowu0162/LongMemEval.

View Paper: https://www.semanticscholar.org/paper/fd48dc6b433cfcd220b6e769c6179d8ef6fcf862

Citations in the corpus

• A Survey on the Memory Mechanism of Large Language Model-based Agents (2024, unknown): “Some previous works have constructed several datasets to support the evaluation of LLM-based agents [26, 191, 194], and some benchmarks also conduct evaluations on the memory capability of LLM-based agents [102, 182, 219].”
• MemStrata: 95% and 90.91% Source-Aware Accuracy on LongMemEval-500 and LoCoMo-1540 with a Local Qwen 3.8 27B Q4_K_M Reader (2026, unknown): “LongMemEval tests multiple forms of long-term interactive memory, including knowledge updates, temporal reasoning, and abstention (Wu et al., 2025).”
• Structured Belief State and the First Precision-Aware Benchmark for LLM Memory Retrieval (2026, unknown): “LLM agent memory benchmarks, such as LongMemEval [1], typically measure downstream answer quality rather than upstream retrieval accuracy.”