Multi-Modal Open-Domain Dialogue

By Kurt Shuster, Eric Michael Smith, Da Ju, Jason Aaron Edward Weston. 2020

49 citations recorded by the source corpus.

Recent work in open-domain conversational agents has demonstrated that significant improvements in humanness and user preference can be achieved via massive scaling in both pre-training data and model size (Adiwardana et al., 2020; Roller et al., 2020). However, if we want to build agents with human-like abilities, we must expand beyond handling just text. A particularly important topic is the ability to see images and communicate about what is perceived. With the goal of getting humans to engage in multi-modal dialogue, we investigate combining components from state-of-the-art open-domain dialogue agents with those from state-of-the-art vision models. We study incorporating different image fusion schemes and domain-adaptive pre-training and fine-tuning strategies, and show that our best resulting model outperforms strong existing models in multi-modal dialogue while simultaneously performing as well as its predecessor (text-only) BlenderBot (Roller et al., 2020) in text-based conversation. We additionally investigate and incorporate safety components in our final model, and show that such efforts do not diminish model performance with respect to human preference.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

“I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022, methodology): “plained by differences in model size and partially by overall differences in generation between the two classes of models (Adiwardana et al., 2020; Roller et al., 2021; Shuster et al., 2021).”
i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Data (2023, methodology): “Multi-Modal BlenderBot(Shuster et al., 2021) The multimodal version of BlenderBot that fuses vision features from ResNet/Faster-RCNN in the multimodal text generation.”
"I'm sorry to hear that": finding bias in language models with a holistic descriptor dataset (2022, methodology): “…has less bias than either of the two BlenderBot 2.0 sizes, which might partially be explained by differences in model size and partially by overall differences in generation performance between the two classes of models (Adiwar-dana et al., 2020; Roller et al., 2021; Shuster et al., 2021).”