Mixture-of-Agents Enhances Large Language Model Capabilities

By Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, James Zou. International Conference on Learning Representations, 2024

450 citations recorded by the source corpus.

Recent advances in large language models (LLMs) demonstrate substantial capabilities in natural language understanding and generation tasks. With the growing number of LLMs, how to harness the collective expertise of multiple LLMs is an exciting open direction. Toward this goal, we propose a new approach that leverages the collective strengths of multiple LLMs through a Mixture-of-Agents (MoA) methodology. In our approach, we construct a layered MoA architecture wherein each layer comprises multiple LLM agents. Each agent takes all the outputs from agents in the previous layer as auxiliary information in generating its response. MoA models achieves state-of-art performance on AlpacaEval 2.0, MT-Bench and FLASK, surpassing GPT-4 Omni. For example, our MoA using only open-source LLMs is the leader of AlpacaEval 2.0 by a substantial gap, achieving a score of 65.1% compared to 57.5% by GPT-4 Omni.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/2b3ad2fdd9d2013119232ee49e6d21eb08474b74

Citations in the corpus

• Large Language Monkeys: Scaling Inference Compute with Repeated Sampling (2024, unknown): “Additionally, multiple models can be ensembled together at inference time to combine their strengths [58, 14, 45, 56, 31].”