Composing Ensembles of Pre-trained Models via Iterative Consensus

By Shuang Li, Yilun Du, Josh Tenenbaum, Antonio Torralba, Igor Mordatch. 2022

33 citations recorded by the source corpus.

Large pre-trained models exhibit distinct and complementary capabilities dependent on the data they are trained on. Language models such as GPT-3 are capable of textual reasoning but cannot understand visual information, while vision models such as DALL-E can generate photorealistic photos but fail to understand complex language descriptions. In this work, we propose a unified framework for composing ensembles of different pre-trained models -- combining the strengths of each individual model to solve various multimodal problems in a zero-shot manner. We use pre-trained models as"generators"or"scorers"and compose them via closed-loop iterative consensus optimization. The generator constructs proposals and the scorers iteratively provide feedback to refine the generated result. Such closed-loop communication enables models to correct errors caused by other models, significantly boosting performance on downstream tasks, e.g. improving accuracy on grade school math problems by 7.5%, without requiring any model finetuning. We demonstrate that consensus achieved by an ensemble of scorers outperforms the feedback of a single scorer, by leveraging the strengths of each expert model. Results show that the proposed method can be used as a general purpose framework for a wide range of zero-shot multimodal tasks, such as image generation, video question answering, mathematical reasoning, and robotic manipulation. Project page: https://energy-based-model.github.io/composing-pretrained-models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Health Informatics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Health Informatics

Citations in the corpus

ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs (2023, background): “Collaborative processes such as brain-storming, group meetings, and discussions play a pivotal role in reaching a consensus and arriving at more refined solutions to complex problems (Li et al., 2022b).”
Improving Factuality and Reasoning in Language Models through Multiagent Debate (2023, background): “Most similar to our work, [14, 32] propose to combine multiple different large pretrained models together for multimodal reasoning.”