LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset

By Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P. Xing and 3 more. International Conference on Learning Representations, 2023

484 citations recorded by the source corpus.

Studying how people interact with large language models (LLMs) in real-world scenarios is increasingly important due to their widespread use in various applications. In this paper, we introduce LMSYS-Chat-1M, a large-scale dataset containing one million real-world conversations with 25 state-of-the-art LLMs. This dataset is collected from 210K unique IP addresses in the wild on our Vicuna demo and Chatbot Arena website. We offer an overview of the dataset's content, including its curation process, basic statistics, and topic distribution, highlighting its diversity, originality, and scale. We demonstrate its versatility through four use cases: developing content moderation models that perform similarly to GPT-4, building a safety benchmark, training instruction-following models that perform similarly to Vicuna, and creating challenging benchmark questions. We believe that this dataset will serve as a valuable resource for understanding and advancing LLM capabilities. The dataset is publicly available at https://huggingface.co/datasets/lmsys/lmsys-chat-1m.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/84a36e19f9394f22b34f79756fa9628a795e02ea

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

HealthBench: Evaluating Large Language Models Towards Improved Human Health (2025, unknown): “These user-model interactions are highly varied and idiosyncratic, and reflect individual user styles in the same way that two people might send emails or text messages in very different styles (Zheng et al., 2024; Zhao et al., 2024).”
WildChat: 1M ChatGPT Interaction Logs in the Wild (2024, background): “Among these, W ILD C HAT and LMSYS-Chat-1M both feature authentic user prompts derived from real user-chatbot interactions, setting them apart from datasets like Alpaca with model-generated prompts, The heatmap shows the average NLLs of fine-tuning Llama-2 7B on one dataset and evaluating NLLs on th”
WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs (2024, methodology): “To cover harm risks in real-world user requests, we include prompts from LMSYS-C HAT - 1M [40] and W ILD C HAT [39] datasets, both of which capture in-the-wild human-LLM interactions, with harm labels assigned based on the OpenAI Moderation API.”
A Survey on Evaluation of Large Language Models (2023, unknown): “[245] have curated a comprehensive LLMs conversation dataset, LMSYS-Chat-1M, encompassing up to onemillion samples.”
Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (2024, background): “A typical LLM prompt contains 100s-1000s of input tokens Table 2, [76].”
Gemma 2: Improving Open Language Models at a Practical Size (2024, methodology): “In particular, we use the prompts, but not the answers from LMSYS-chat-1M (Zheng et al., 2023).”
Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing (2024, methodology): “To create datasets for alignment, previous studies have collected human interactions with LLMs [14, 64, 65, 66, 26].”
CUEing User Simulators: Calibrated User Embeddings for Multi-Turn Benchmarking (2026, unknown): “We use English multi-turn conversations from 24 DialogStudio datasets (Zhang et al., 2024a), LMSYS-Chat-1M (Zheng et al., 2024), and WildChat-1M (Zhao et al., 2024), balancing by capping each dataset at 10k samples.”
Dataset Featurization: Uncovering Natural Language Features through Unsupervised Data Reconstruction (2025, unknown): “• LMSYS-CHAT-1M (Zheng et al., 2023b): LMSYS-Chat-1M Dataset License Agreement .”