All-in-One Image-Grounded Conversational Agents

By Da Ju, Kurt Shuster, Y-Lan Boureau, Jason Aaron Edward Weston. 2019

9 citations recorded by the source corpus.

As single-task accuracy on individual language and image tasks has improved substantially in the last few years, the long-term goal of a generally skilled agent that can both see and talk becomes more feasible to explore. In this work, we focus on leveraging individual language and image tasks, along with resources that incorporate both vision and language towards that objective. We design an architecture that combines state-of-the-art Transformer and ResNeXt modules fed into a novel attentive multimodal module to produce a combined model trained on many tasks. We provide a thorough analysis of the components of the model, and transfer performance when training on one, some, or all of the tasks. Our final models provide a single system that obtains good results on all vision and language tasks considered, and improves the state-of-the-art in image-grounded conversational applications.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Dialogue & Conversational AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Dialogue & Conversational AI

Citations in the corpus

Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “Recent work also shows that we can combine both nonconversational multimodal data and conversational multimodal data to obtain strong performance on both (Ju et al., 2019).”
Multi-Modal Open-Domain Dialogue (2020, methodology): “• 2AMMC (Ju et al., 2019), in which multiple Transformers are attended over in order to make use of a combination of ResNeXt-IG-3.”
i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Data (2023, methodology): “2AMMC (Ju et al., 2019) A multimodal generative model that combines ResNet and text transformer.”