MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark

By Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, Yashar Mehdad. 2020

223 citations recorded by the source corpus.

Scaling semantic parsing models for task-oriented dialog systems to new languages is often expensive and time-consuming due to the lack of available datasets. Available datasets suffer from several shortcomings: a) they contain few languages b) they contain small amounts of labeled examples per language c) they are based on the simple intent and slot detection paradigm for non-compositional queries. In this paper, we present a new multilingual dataset, called MTOP, comprising of 100k annotated utterances in 6 languages across 11 domains. We use this dataset and other publicly available datasets to conduct a comprehensive benchmarking study on using various state-of-the-art multilingual pre-trained models for task-oriented semantic parsing. We achieve an average improvement of +6.3 points on Slot F1 for the two existing multilingual datasets, over best results reported in their experiments. Furthermore, we demonstrate strong zero-shot performance using pre-trained models combined with automatic translation and alignment, and a proposed distant supervision method to reduce the noise in slot label projection.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

UL2: Unifying Language Learning Paradigms (2022, background): “We use several component tasks from UnifiedSKG (Xie et al., 2022), namely WikiTQ (Pasupat & Liang, 2015), CompWQ (Talmor & Berant, 2018), FetaQA (Nan et al., 2021), HybridQA (Chen et al., 2020), WikiSQL (Zhong et al., 2017), TabFat (Chen et al., 2019), Feverous (Aly et al., 2021), SQA (Iyyer et al.,”
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022, background): “, 2016) Knowledge Graph Question s-Expression MTOP (Li et al., 2021) API Calls Question TOP Representation”
Learning To Retrieve Prompts for In-Context Learning (2021, methodology): “• MTOP (Li et al., 2021) is a recent semantic parsing dataset, focused on task-oriented dialogues, where commands are mapped to complex nested queries across 11 domains, such as alarm, messaging, music, recipes, etc.”
Unifying Language Learning Paradigms (2022, unknown): “We use several component tasks from UniﬁedSKG (Xie et al., 2022), namely WikiTQ (Pasupat & Liang, 2015), CompWQ (Talmor & Be-rant, 2018), FetaQA (Nan et al., 2021), HybridQA (Chen et al., 2020), WikiSQL (Zhong et al., 2017), TabFat (Chen et al., 2019), Feverous (Aly et al., 2021), SQA (Iyyer et al.,”
NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (2024, unknown): “…(O’Neill et al., 2021), Banking77 (Casanueva et al., 2020), Emotion (Saravia et al., 2018), IMDB (Maas et al., 2011), MTOPDomain/MTOPIntent (Li et al., 2021), ToxicConversations (Adams et al., 2019), TweetSentimentExtraction (Maggie, 2020), AmazonPolarity (McAuley & Leskovec, 2013b),…”
MTEB: Massive Text Embedding Benchmark (2022, background): “(d) Reproducibility: Through versioning at a dataset and software level, we aim to make it easy to repro- MTEB 8 Tasks 58 Datasets Massive Text Embedding Benchmark Classification AmazonCounterfactual Retrieval Pair Classification AmazonPolarity AmazonReviews Banking77 Emotion Imdb MassiveIntent Mass”
Controllable Semantic Parsing via Retrieval Augmentation (2021, methodology): “Data preprocessing We used the entire English portion of the MTOP dataset (Li et al., 2021a .”
Language Models are Few-shot Multilingual Learners (2021, methodology): “…et al., 2021; Lee et al., 2020; Cahyawijaya et al., 2021b; Park et al., 2021) and then apply pre-training on the source language and ﬁne-tune on the target languages (Schuster et al., 2019; Lin et al., 2019; Winata et al., 2019, 2021; Pfeiffer et al., 2020; Zheng et al., 2021; Lin et al., 2021b).”
MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages (2022, background): “, 2021), or contain natural language intents collected via automatic translation (Li et al., 2020).”
BiToD: A Bilingual Multi-Domain Dataset For Task-Oriented Dialogue Modeling (2021, background): “In particular, Multi-SLU [46], Multi-ATIS [47], and MTOP [48] are multilingual datasets designed to study cross-lingual transfer learning for SLU.”
nmT5 - Is parallel data still relevant for pre-training massively multilingual language models? (2021, background): “• MTOP (Li et al., 2020) prediction, where user queries must be parsed into a tree, capturing the domain, intent and slots.”
MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages (2022, unknown): “, 2018) 1 44,873 2 25 36 Multilingual Task-Oriented Semantic Parsing (MTOP) (Li et al., 2021) 6 15,195-22,288 11 104-113 72-75”
Layer-wise Representation Dynamics: An Empirical Investigation Across Embedders and Base LLMs (2026, unknown): “…MassiveIntentClassification [16] Classification Accuracy MassiveScenarioClassification [16] Classification Accuracy MTOPDomainClassification [36] Classification Accuracy MTOPIntentClassification [36] Classification Accuracy ToxicConversationsClassification Classification Accuracy…”
Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation (2025, unknown): “• MTOPIntent (Li et al., 2020): MTOP is a multilingual dataset introduced in 2021.”