Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language

By Andy Zeng, Adrian S. Wong, Stefan Welker, Krzysztof Marcin Choromanski, Federico Tombari, Aveek Purohit, Michael S. Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke and 1 more. 2022

742 citations recorded by the source corpus.

Large pretrained (e.g.,"foundation") models exhibit distinct capabilities depending on the domain of data they are trained on. While these domains are generic, they may only barely overlap. For example, visual-language models (VLMs) are trained on Internet-scale image captions, but large language models (LMs) are further trained on Internet-scale text with no images (e.g., spreadsheets, SAT questions, code). As a result, these models store different forms of commonsense knowledge across different domains. In this work, we show that this diversity is symbiotic, and can be leveraged through Socratic Models (SMs): a modular framework in which multiple pretrained models may be composed zero-shot i.e., via multimodal-informed prompting, to exchange information with each other and capture new multimodal capabilities, without requiring finetuning. With minimal engineering, SMs are not only competitive with state-of-the-art zero-shot image captioning and video-to-text retrieval, but also enable new applications such as (i) answering free-form questions about egocentric video, (ii) engaging in multimodal assistive dialogue with people (e.g., for cooking recipes) by interfacing with external APIs and databases (e.g., web search), and (iii) robot perception and planning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Emergent Abilities of Large Language Models (2022, neutral): “…via prompting to translate natural language instructions into actions executable by robots (Ahn et al., 2022; Huang et al., 2022), interact with users (Coenen et al., 2021; Wu et al., 2021; 2022a; Lee et al., 2022b), and facilitate multi-modal reasoning (Zeng et al., 2022; Alayrac et al., 2022).”
Augmented Language Models: a Survey (2023, methodology): “Zeng et al. (2022) introduce Socratic Models, a modular framework in which various models pre-trained on different modalities can be composed zero-shot.”
Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, methodology): “…for (a) mobile manipulation and (b,c) tabletop manipulation in both simulated and real settings. multimodal-informed perception through language [19], to the best of our knowledge no work has studied the critical link of not only planning with language, but also informing embodied feedback with…”
Language Model Cascades (2022, background): “Socratic models (Zeng et al., 2022) extends model chaining to the multimodal setting and demonstrates zero-shot abilities on tasks for which no single model exists.”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “One recent line of work [26, 68, 78, 114, 136, 144] proposes to freeze the pretrained LM weights to prevent catastrophic forgetting [71].”
ViperGPT: Visual Inference via Python Execution for Reasoning (2023, background): “Other recent works [63, 45, 55, 35, 37, 15] show that large pretrained models can be used together to great effect, but hand-specify the particular way models are combined.”
Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models (2023, background): “With the increased number of model parameters, it is hard to train those pretrained LLMs, thus more efforts have been paid to directly leverage the off-the-shelf frozen pre-trained LLMs for VL tasks [12, 38, 8, 46, 50].”
PaLM-E: An Embodied Multimodal Language Model (2023, methodology): “…world models (Nottingham et al., 2023; Zellers et al., 2021a), planning over graphs and maps (Shah et al., 2022; Huang et al., 2022a), visual explanations (Wang et al., 2023), program generation (Liang et al., 2022; Singh et al., 2022), or injecting information into the prompt (Zeng et al., 2022).”
Visual Programming: Compositional visual reasoning without training (2022, background): “Socratic models (SMs) [36], compose pretrained models from different modalities such as language (BERT [7], GPT-2 [24]), vision-language (CLIP [23]), and audio-language (mSLAM [3]), to perform a number of zero-shot tasks, including image captioning, video-to-text retrieval, and robot planning.”
CoCa: Contrastive Captioners are Image-Text Foundation Models (2022, methodology): “Using code3 provided by the authors of Socratic Models [70], we re-computed metrics on the available subset for those methods, indicated by “(subset)” for fairest comparison.”
Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners (2022, methodology): “Concurrent work Socratic [68] uses a zero-shot language-based world-state history to represent long videos with given time stamps, while our model can quickly adapt to different video and text distributions with few examples.”
Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training (2022, background): “Zeng et al. (2022a) show that language-conjoined LM and VLM successfully perform captioning and retrieval but do not evaluate their models on VQA.”
Open-vocabulary Queryable Scene Representations for Real World Planning (2022, background): “Recently LLM-based planners are more flexible [6], [7], [36] and do not require handcrafting predicates, however, they do not handle the complexity of open-vocabulary object proposal and require defining a set of objects involved in planning.”
Contrastive Distillation Is a Sample-Efficient Self-Supervised Loss Policy for Transfer Learning (2022, neutral): “, 2022) and to answer questions for planning in robotics (Zeng et al., 2022).”
Improving Factuality and Reasoning in Language Models through Multiagent Debate (2023, neutral): “Most similar to our work, (Li et al., 2022a; Zeng et al., 2022) pro-pose to combine multiple different large pretrained models together for multimodal reasoning.”
ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs (2023, background): “This has led to the development of ensembles (Sagi & Rokach, 2018) in multimodal learning (Zeng et al., 2023; Li et al., 2022a).”