Evaluating Very Long-Term Conversational Memory of LLM Agents

By Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, Yuwei Fang. Annual Meeting of the Association for Computational Linguistics, 2024

642 citations recorded by the source corpus.

Existing works on long-term open-domain dialogues focus on evaluating model responses within contexts spanning no more than five chat sessions. Despite advancements in long-context large language models (LLMs) and retrieval augmented generation (RAG) techniques, their efficacy in very long-term dialogues remains unexplored. To address this research gap, we introduce a machine-human pipeline to generate high-quality, very long-term dialogues by leveraging LLM-based agent architectures and grounding their dialogues on personas and temporal event graphs. Moreover, we equip each agent with the capability of sharing and reacting to images. The generated conversations are verified and edited by human annotators for long-range consistency and grounding to the event graphs. Using this pipeline, we collect LoCoMo, a dataset of very long-term conversations, each encompassing 300 turns and 9K tokens on avg., over up to 35 sessions. Based on LoCoMo, we present a comprehensive evaluation benchmark to measure long-term memory in models, encompassing question answering, event summarization, and multi-modal dialogue generation tasks. Our experimental results indicate that LLMs exhibit challenges in understanding lengthy conversations and comprehending long-range temporal and causal dynamics within dialogues. Employing strategies like long-context LLMs or RAG can offer improvements but these models still substantially lag behind human performance.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/0bf3a1867f7245b8a702093901c66b08b518eafc

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Juntao Tan
2. Kai Mei
3. Kai-Wei Chang
4. Nanyun Peng
5. Mohit Bansal
6. Adina Williams
7. Yulia Tsvetkov
8. Yejin Choi

Citations in the corpus

• A Survey on the Memory Mechanism of Large Language Model-based Agents (2024, neutral): “Some previous works have constructed several datasets to support the evaluation of LLM-based agents [26, 191, 194], and some benchmarks also conduct evaluations on the memory capability of LLM-based agents [102, 182, 219].”
• A-MEM: Agentic Memory for LLM Agents (2025, positive): “Through analysis of two dialogues sampled from long-term conversations in LoCoMo[22], we observe that A-M EM (shown in blue) consistently produces more coherent clustering patterns compared to the baseline system (shown in red).”
• LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory (2024, positive): “LoCoMo (Maharana et al., 2024) includes 50 long-term chat histories and questions testing single-hop, multi-hop, temporal, commonsense, world knowledge, and adversarial reasoning.”
• Choosing How to Remember: Adaptive Memory Structures for LLM Agents (2026, unknown): “To evaluate the effectiveness of our proposed framework, we conduct a series of experiments across two representative datasets - LoCoMo (Maharana et al., 2024) and PER-SONAMEM (Jiang et al., 2025).”
• eMEM: A Hybrid Spatio-Temporal Memory System For Embodied Agents (2026, unknown): “Lo-CoMo [Maharana et al., 2024] and similar long-horizon dialogue benchmarks evaluate text-only conversational memory and do not exercise spatial grounding, multi-modal perception, or interoception; the axes that motivate embodied memory in the first place.”
• NEAR: Neighborhood-Aware Residuals for Memory Retrieval (2026, unknown): “We evaluate retrieval on eight benchmarks: LoCoMo (Maharana et al., 2024), Long-MemEval (Wu et al., 2025), and REALTALK (Lee et al., 2025) for conversational memory, and ESG-Reports (Macé et al., 2025), QASPER (Dasigi et al., 2021), PeerQA (Baumgärtner et al., 2025), CovidQA (Möller et al., 2020), a”
• MemStrata: 95% and 90.91% Source-Aware Accuracy on LongMemEval-500 and LoCoMo-1540 with a Local Qwen 3.8 27B Q4_K_M Reader (2026, unknown): “We evaluate this configuration on the 500-question LongMemEval-S set (Wu et al., 2025), called LongMemEval-500 (LME500) in the title, and on 1,540 category 1–4 questions in the ten-history LoCoMo dataset (Maharana et al., 2024), called LoCoMo-1540 in the title.”
• Structured Belief State and the First Precision-Aware Benchmark for LLM Memory Retrieval (2026, unknown): “Like LoCoMo [4], StructMemEval assesses correctness at the answer layer rather than directly scoring the retrieved set.”
• ER-MIA: Black-Box Adversarial Memory Injection Attacks on Long-Term Memory-Augmented Large Language Models (2026, unknown): “…to model parameters or the memory system internals. has proposed equipping LLMs with external long-term memory systems that store information from past interactions in memory banks and retrieve relevant memories during inference (Xu et al., 2025; Chhikara et al., 2025; Maharana et al., 2024).”
• Mindscape-Aware Retrieval Augmented Generation for Improved Long Context Understanding (2025, unknown): “We further validate this generalization on the long-term dialogue benchmark LoCoMo (Maharana et al., 2024) in Appx.”