ParlAI: A Dialog Research Software Platform

By Alexander H. Miller, Will Feng, Dhruv Batra, Antoine Bordes, Adam Fisch, Jiasen Lu, Devi Parikh, Jason Aaron Edward Weston. 2017

401 citations recorded by the source corpus.

We introduce ParlAI (pronounced “par-lay”), an open-source software platform for dialog research implemented in Python, available at http://parl.ai. Its goal is to provide a unified framework for sharing, training and testing dialog models; integration of Amazon Mechanical Turk for data collection, human evaluation, and online/reinforcement learning; and a repository of machine learning models for comparing with others’ models, and improving upon existing architectures. Over 20 tasks are supported in the first release, including popular datasets such as SQuAD, bAbI tasks, MCTest, WikiQA, QACNN, QADailyMail, CBT, bAbI Dialog, Ubuntu, OpenSubtitles and VQA. Several models are integrated, including neural models such as memory networks, seq2seq and attentive LSTMs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2018, background): “The recently proposed dialogue systems framework ParlAI (Miller et al., 2017) also combines many language understanding tasks into a single framework, although this aggregation is very flexible, and the framework includes no standardized evaluation suite for system performance.”
CoQA: A Conversational Question Answering Challenge (2018, methodology): “These measures help to prevent spam and to obtain high agreement data.4 We use Amazon Mechanical Turk to pair workers on a passage through the ParlAI MTurk API (Miller et al., 2017).”
fairseq: A Fast, Extensible Toolkit for Sequence Modeling (2019, background): “, 2018b, 2019), error correction (Chollampatt and Ng, 2018), multilingual sentence embeddings (Artetxe and Schwenk, 2018), and dialogue (Miller et al., 2017; Dinan et al., 2019).”
“I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022, methodology): “Prepending these persona strings to the HOLISTICBIAS templated sentence prompt allows for a greater diversity of possible responses by the generative model.4 We perform generations using the ParlAI framework (Miller et al., 2017).”
Recipes for Safety in Open-domain Chatbots (2020, methodology): “Models were fine-tuned using the ParlAI toolkit (Miller et al., 2017a).”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, methodology): “Our ParlAI (Miller et al., 2017)5 framework attempts to integrate standardized data, released models and software as a one-stop solution and its wide adoption by the community has created an ecosystem of improved support, feature development, and engineering improvements.”
Finding Generalizable Evidence by Learning to Convince Q&A Models (2019, methodology): “uations, we use Amazon Mechanical Turk via ParlAI (Miller et al., 2017).”
Retrieval Augmentation Reduces Hallucination in Conversation (2021, methodology): “Training Details All models are trained in ParlAI3 (Miller et al., 2017), sweeping over parameters where possible, and using early stopping of model perplexity on the validation set.”
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering (2018, methodology): “We adapt ParlAI for collecting question answer pairs by converting the collection workﬂow into a system-oriented dialog.”
Learning from Dialogue after Deployment: Feed Yourself, Chatbot! (2019, methodology): “The datasets and models described in this paper are available via the ParlAI platform (Miller et al., 2017), along with training code.”
Queens Are Powerful Too: Mitigating Gender Bias in Dialogue Generation (2019, methodology): “All models are 8-layer encoders, 8-layer decoders, with 512 dimensional embeddings and 16 attention heads based on the ParlAI transformer implementation (Miller et al., 2017).”
Does Gender Matter? Towards Fairness in Dialogue Systems (2019, methodology): “The models are trained under the ParlAI framework (Miller et al., 2017).”
Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators (2019, methodology): “We pretrained models that produce sentence embeddings using the ParlAI chitchat data set (Miller et al., 2017).”
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage (2022, methodology): “A critical part of the program is that, as much as possible, the research should be accessible and reproducible (Roller et al., 2020; Miller et al., 2017).”
KILT: a Benchmark for Knowledge Intensive Language Tasks (2020, methodology): “We use the pre-trained version available in ParlAI (Miller et al., 2017).”
Recipes for Building an Open-Domain Chatbot (2020, methodology): “Our implementation is based on the ParlAI version (Miller et al., 2017).”
Bot-Adversarial Dialogue for Safe Conversational Agents (2021, methodology): “We consider the same architecture and setup as in BlenderBot (Roller et al., 2020), which employs a Seq2Seq Trans-former architecture (Vaswani et al., 2017), with an implementation based on the ParlAI version (Miller et al., 2017a).”
Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments (2017, methodology): “In addition to the C++ API, Python bindings are also provided, allowing the simulator to be easily used with deep learning frameworks such as Caffe [20] and TensorFlow [1], or within RL platforms such as ParlAI [33] and OpenAI Gym [8].”
Using In-Context Learning to Improve Dialogue Safety (2023, methodology): “We use ParlAI (Miller et al., 2017) 12 We use all of the available prompts for detoxification.”
AllenNLP: A Deep Semantic Natural Language Processing Platform (2018, background): “Finally, AllenNLP is related to toolkits for deep learning research in dialog (Miller et al., 2017) and machine translation (Klein et al., 2017).”