Large Language Model Evaluation Via Multi AI Agents: Preliminary results

Highlighting the growing significance of LLMs in both academia and industry, this multi-agent AI model offers a dedicated system for evaluating and comparing the performance of various LLMs comprehensively.

• Eight AI agents deployed for code retrieval based on a common description from different LLMs including GPT-3.5, GPT-4, and others.
• Verification agent assesses code against the HumanEval benchmark to determine performance quality.
• GPT-3.5 Turbo showcased better performance compared to other models in initial tests.
• Plans to include MBPP benchmark for an enhanced evaluation framework.

As LLMs become increasingly pivotal, the necessity for robust evaluation frameworks is undeniable. This novel approach to benchmarking such models provides critical insights into their capabilities, guiding further development and ensuring responsible and impactful integration of these models across various sectors.