QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization

By Ming Zhong, Da Yin, Tao Yu, Ahmad Zairi Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu and 1 more. 2021

488 citations recorded by the source corpus.

Meetings are a key component of human collaboration. As increasing numbers of meetings are recorded and transcribed, meeting summaries have become essential to remind those who may or may not have attended the meetings about the key decisions made and the tasks to be completed. However, it is hard to create a single short summary that covers all the content of a long meeting involving multiple people and topics. In order to satisfy the needs of different types of users, we define a new query-based multi-domain meeting summarization task, where models have to select and summarize relevant spans of meetings in response to a query, and we introduce QMSum, a new benchmark for this task. QMSum consists of 1,808 query-summary pairs over 232 meetings in multiple domains. Besides, we investigate a locate-then-summarize method and evaluate a set of strong summarization baselines on the task. Experimental results and manual analysis reveal that QMSum presents significant challenges in long meeting summarization for future research. Dataset is available at https://github.com/Yale-LILY/QMSum.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

SCROLLS: Standardized CompaRison Over Long Language Sequences (2022, methodology): “Figures 5, 6, 7, 8 and 9 showcase gold references, relevant parts from input documents required to generate those references, and queries when exist, from GovReport, QMSum, Qasper, NarrativeQA and ContractNLI.”
UL2: Unifying Language Learning Paradigms (2022, methodology): “, 2021), QMSUm (Zhong et al., 2021), QASPER (Dasigi et al.”
Survey of Hallucination in Natural Language Generation (2022, background): “[235] show that 74% of samples in the QMSum dataset consist of inconsistent facts.”
A Length-Extrapolatable Transformer (2022, methodology): “We evaluate models on PG22 and QMSum (Zhong et al., 2021) with various input lengths.”
CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuning (2021, background): “…of dialogue summarization has attracted signiﬁcant research attentions (Mc-Cowan et al., 2005; Gliwa et al., 2019; Koay et al., 2020; Zhang et al., 2021; Zhong et al., 2021; Zhu et al., 2021; Chen et al., 2021a; Li et al., 2021; Chen et al., 2021c; Fabbri et al., 2021; Chen et al., 2021d).”
InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning (2022, background): “Going beyond multi-task training, these approaches show better generalization to unseen tasks when prompted with a few examples (Bragg et al., 2021; Min et al., 2022a,b) or language definitions and constraints (Weller et al., 2020; Zhong et al., 2021b; Xu et al., 2022).”
SnapKV: LLM Knows What You are Looking for Before Generation (2024, background): “To further validate this finding, we also observe on multiple long documents QA datasets including QMSum [12], a query-based multi-domain meeting summarization; Openreview [13], a collection of papers from openreview.net ; SPACE [14], an extractive opinion summarization in quantized transformer…”
LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks (2024, unknown): “…tasks, including document QA (Koˇcisk`y et al., 2018; Dua et al., 2019; Dasigi et al., 2021; Pang et al., 2022; Wang et al., 2024a), summarization (Zhong et al., 2021; Huang et al., 2021; Wang et al., 2022), retrieval and attributing (Kamradt, 2023; Kuratov et al., 2024; Song et al., 2025; Laban…”
Generative artificial intelligence: a systematic review and applications (2024, unknown): “Additionally, the outcomes of PGnet [136], BART[75], HMNet[143], and ChatGPT[15] were assessed using the QMSUM dataset[156].”
Retentive Network: A Successor to Transformer for Large Language Models (2023, background): “(Zhong et al., 2021), GovReport (Huang et al., 2021), SummScreen (Chen et al., 2021; Shaham et al., 2022”
PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling (2024, background): “It includes 17 datasets covering tasks such as single-document QA [20, 6], multi-document QA [33, 16], summarization [17, 35, 10], few-shot learning [21, 13, 19], synthetic, and code generation [14, 24].”
LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (2023, methodology): “We also sampled from QMSum (Zhong et al., 2021), which consists of query-summary pairs annotated over 232 meetings across multiple domains, including product, academic, and committee meetings.”
RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models (2023, background): “In different contexts, the definition and criteria for hallucination vary, and the annotation of hallucina-310 tion is not always straightforward.”
Multi-document Summarization via Deep Learning Techniques: A Survey (2020, methodology): “The representative dialogue summarization datasets are DIALOGSUM [25], AMI [23], MEDIASUM [185], and QMSum [183].”
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2022, background): “Reference summaries can be viewed as an implicit way of providing such specifications, but one can also explicitly provide a specification such as an information need in the form of a query (Zhong et al., 2021).”
Gated Delta Networks: Improving Mamba2 with Delta Rule (2024, unknown): “…et al., 2018), 2WikiMulti QA (Ho et al., 2020), Musique (Trivedi et al., 2022)), document summarization (GovReport (Huang et al., 2021), QMSum (Zhong et al., 2021), MultiNews (Fabbri et al., 2019)), and various specialized tasks (TRec (Li & Roth, 2002), Trivia QA (Joshi et al., 2017b), SamSum…”
Effective Long-Context Scaling of Foundation Models (2023, methodology): “We evaluate 0-shot performance on NarrativeQA (Koˇciský et al., 2018), 2-shot on QuALITY (Pang et al., 2022) and Qasper (Dasigi et al., 2021), and 1-shot on QMSum (Zhong et al., 2021).”
Unifying Language Learning Paradigms (2022, methodology): “…(Shaham et al., 2022) which comprises of seven component tasks including GovReport (Huang et al., 2021), SumScr (Chen et al., 2021), QMSUm (Zhong et al., 2021), QASPER (Dasigi et al., 2021), NarrativeQA (Ko ˇ cisk ´ y et al., 2018), QuaLITY (Pang et al., 2021), and ContractNLI (Koreeda &…”