ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs
By J. Chen, Swarnadeep Saha, Mohit Bansal. Annual Meeting of the Association for Computational Linguistics, 2023
336 citations recorded by the source corpus.
Large Language Models (LLMs) still struggle with natural language reasoning tasks. Motivated by the society of minds (Minsky, 1988), we propose ReConcile, a multi-model multi-agent framework designed as a round table conference among diverse LLM agents. ReConcile enhances collaborative reasoning between LLM agents via multiple rounds of discussion, learning to convince other agents to improve their answers, and employing a confidence-weighted voting mechanism that leads to a better consensus. In each round, ReConcile initiates discussion between agents via a 'discussion prompt' that consists of (a) grouped answers and explanations generated by each agent in the previous round, (b) their confidence scores, and (c) demonstrations of answer-rectifying human explanations, used for convincing other agents. Experiments on seven benchmarks demonstrate that ReConcile significantly improves LLMs' reasoning -- both individually and as a team -- surpassing prior single-agent and multi-agent baselines by up to 11.4% and even outperforming GPT-4 on three datasets. ReConcile also flexibly incorporates different combinations of agents, including API-based, open-source, and domain-specific models, leading to an 8% improvement on MATH. Finally, we analyze the individual components of ReConcile, demonstrating that the diversity originating from different models is critical to its superior performance. Code: https://github.com/dinobby/ReConcile
Subfields: Artificial Intelligence
View Paper: https://www.semanticscholar.org/paper/63549bf78e4b1e7e1cec505ce65e6e8f90474f41
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Xinyun Chen
- Denny Zhou
- Swaroop Mishra
- Jie Huang
- Adams Wei Yu
- Jue Wang
- Junlin Wang
- Luke S. Zettlemoyer
Citations in the corpus
- Large Language Models Cannot Self-Correct Reasoning Yet (2023, methodology): “, 2023; Chen et al., 2023a). Du et al. (2023) implement a multi-agent debate method by leveraging multiple instances of a single ChatGPT model and demonstrate significant improvements on reasoning tasks. We adopt their method to test performance on GSM8K. For an unbiased implementation, we use the e”
- Mixture-of-Agents Enhances Large Language Model Capabilities (2024, background): “Moreover, ReConcile (Chen et al., 2023a) exemplifies an asymmetric discussion involving weighted voting.”
- Empowering Biomedical Discovery with AI Agents (2024, neutral): “Different strategies are used for multi-agent interaction, including cooperation [129–131] and negotiation [73, 132, 133].”