Mixture-of-Agents Enhances Large Language Model Capabilities

Recent advances in large language models (LLMs) demonstrate substantial capabilities in natural language understanding and generation tasks. With the growing number of LLMs, how to harness the collective expertise of multiple LLMs is an exciting open direction. Toward this goal, we propose a new approach that leverages the collective strengths of multiple LLMs through a Mixture-of-Agents (MoA) methodology. In our approach, we construct a layered MoA architecture wherein each layer comprises multiple LLM agents. Each agent takes all the outputs from agents in the previous layer as auxiliary information in generating its response. MoA models achieve state-of-art performance on AlpacaEval 2.0, MT-Bench, and FLASK, surpassing GPT-4 Omni. For example, our MoA using only open-source LLMs is the leader of AlpacaEval 2.0 by a substantial gap, achieving a score of 65.1% compared to 57.5% by GPT-4 Omni.

Key Points:

• Proposed approach leverages multiple LLMs through MoA methodology
• Construction of a layered MoA architecture for improved performance
• Outperformed GPT-4 Omni on AlpacaEval 2.0, MT-Bench, and FLASK tasks
• Open-source MoA led AlpacaEval 2.0 with a score of 65.1%

Opinion:

The integration of Mixture-of-Agents in LLMs marks a significant advancement in natural language understanding and generation tasks. This approach opens up avenues for collaborative learning among LLM agents, paving the way for more sophisticated language models in the future.