Cross-lingual Transfer Learning for Multilingual Task Oriented Dialog

By Sebastian Schuster, Sonal Gupta, Rushin Shah, Michael Lewis. 2018

321 citations recorded by the source corpus.

One of the first steps in the utterance interpretation pipeline of many task-oriented conversational AI systems is to identify user intents and the corresponding slots. Since data collection for machine learning models for this task is time-consuming, it is desirable to make use of existing data in a high-resource language to train models in low-resource languages. However, development of such models has largely been hindered by the lack of multilingual training data. In this paper, we present a new data set of 57k annotated utterances in English (43k), Spanish (8.6k) and Thai (5k) across the domains weather, alarm, and reminder. We use this data set to evaluate three different cross-lingual transfer methods: (1) translating the training data, (2) using cross-lingual pre-trained embeddings, and (3) a novel method of using a multilingual machine translation encoder as contextual word representations. We find that given several hundred training examples in the the target language, the latter two methods outperform translating the training data. Further, in very low-resource settings, multilingual contextual word representations give better results than using cross-lingual static embeddings. We also compare the cross-lingual methods to using monolingual resources in the form of contextual ELMo representations and find that given just small amounts of target language data, this method outperforms all cross-lingual methods, which highlights the need for more sophisticated cross-lingual methods.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

WikiLingua: A New Benchmark Dataset for Multilingual Abstractive Summarization (2020, methodology): “…the English training articles into the target language (DC+Synth) , using AWS Translate, and pairing them with the original summary in English. data has been shown to be an effective strategy for cross-lingual transfer for text classiﬁcation and sequence labeling tasks (Schuster et al., 2019).”
Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation (2019, background): “• We outperform the state-of-the-art on zero-a shot cross-lingual POS tagging [Universal Dependencies 2.3 dataset (Nivre et al., 2018)], intent classiﬁcation (Schuster et al., 2018), and achieve results comparable to state-of-the-art on document classiﬁcation [ML-Doc dataset (Schwenk and Li, 2018)].”
Out-of-Domain Detection for Natural Language Understanding in Dialog Systems (2019, methodology): “We evaluated our method on two datasets: (1) real world data that were collected and annotated from the user logs of Bixby: an intelligent personal assistant (IPA) developed by Samsung electronics; (2) the Facebook multilingual Task Oriented (FTO) dataset (Schuster et al., 2018).”
MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages (2022, background): “Cross-lingual Multilingual Task Oriented Dialog (Schuster et al., 2019) 3 5,083-43,323 3 12 11 Microsoft Dialog Challenge (Li et al.”
Cross-Lingual Transfer Learning for Question Answering (2019, background): “Using MT to achieve knowledge transfer between different languages has been studied on sentiment analysis (Moham-mad et al., 2016), spoken language understanding (Stepanov et al., 2013; Schuster et al., 2018) and question answer (Asai et al., 2018).”
Language Models are Few-shot Multilingual Learners (2021, methodology): “In another stream of work, Shin et al. (2020); Li and Liang (2021) proposed an automated method to create prompts for a diverse set of tasks by gradient-based tuning instead of manually searching for a good prompt.”
An Empirical Survey of Data Augmentation for Limited Data Learning in NLP (2021, background): “To this end, researchers have adopted cross-lingual transfer learning which transfers models from resource-rich source to resource-poor target (Do and Gaspers, 2019; Lee and Lee, 2019; Schuster et al., 2019), few/zero-shot learning (Johnson et al.”
XQA: A Cross-lingual Open-domain Question Answering Dataset (2019, background): “These models have shown their effectiveness in several cross-lingual NLP tasks such as document classiﬁcation (Klementiev et al., 2012), textual similarity (Cer et al., 2017), natural language inference (Conneau et al., 2018b), and dialog sys-tem (Schuster et al., 2019).”
Multilingual and Cross-Lingual Intent Detection from Spoken Data (2021, background): “…technology to other languages has been recognised only recently, and thus multi-lingual intent detection datasets are still few and far between: Schuster et al. (2019) provide NLU data in three languages (English, Spanish, Thai), while a more recent MultiATIS++ dataset (Xu et al., 2020)…”
Teaching a New Dog Old Tricks: Resurrecting Multilingual Retrieval Using Zero-Shot Learning (2019, methodology): “retrieval models using English training data. In the past few years, transfer learning between languages has been proven to be a remarkably eﬀective approach for low-resource multilingual tasks (e.g. [16,17,29,38]). Our model leverages a pre-trained multi-language transformer model to obtain an enco”
Towards multidomain and multilingual abusive language detection: a survey (2021, background): “Cross-lingual transfer learning is the common approach to transfer knowledge from one language (usually with more available resources) to another language (usually with less resources) [69, 126].”
BiToD: A Bilingual Multi-Domain Dataset For Task-Oriented Dialogue Modeling (2021, background): “[46] Sebastian Schuster, Sonal Gupta, Rushin Shah, and Mike Lewis.”
Cross-Lingual Training for Automatic Question Generation (2019, background): “(Similarly in spirit, cross-lingual transfer learning based approaches have been recently proposed for other NLP tasks such as machine translation (Schuster et al., 2019; Lample and Conneau, 2019).”
SLURP: A Spoken Language Understanding Resource Package (2020, unknown): “1, similar to annotations used in (Budzianowski et al., 2018; Schuster et al., 2019).”
Zero-shot Cross-lingual Dialogue Systems with Transferable Latent Variables (2019, methodology): “The main approach of previous work (Upadhyay et al., 2018; Chen et al., 2018; Schuster et al., 2019) in this task is using aligned cross-lingual word embeddings between source and target languages.”
CONAN - COunter NArratives through Nichesourcing: a Multilingual Dataset of Responses to Fight Online Hate Speech (2019, background): “With parallel corpora we can exploit cross-lingual word embeddings to enable knowledge transfer between languages (Schuster et al., 2018).”
Learning Knowledge Bases with Parameters for Task-Oriented Dialogue Systems (2020, background): “, 2020) and target languages (Mrkšić et al., 2017; Schuster et al., 2019; Chen et al., 2018; Liu et al., 2019b), and large pre-trained language models are shown to possess the capability to quickly adapt to taskoriented dialogue tasks by using only a few data samples (Peng et al.”