This paper introduces a paradigm shift from static to dynamic evaluation of AI assistants’ API invocation capabilities using a framework called Automated Dynamic Evaluation (AutoDE). Developed to simulate real human interaction patterns without actual human involvement, this assessment method aims to offer more accurate insights into AI performances.

• AutoDE Framework: Endeavors to mimic human conversation patterns.
• Experimental Setup: Tested on four AI assistants.
• Advantages: Uncovers errors missed by static evaluations, closer alignment with human assessment.
• Human-Like Interaction: Utilizes a LLM-based user agent with scripted interactions.
• Future Application: Potential to innovate human-machine interaction research.

This innovative approach to evaluation redefines how we test and validate AI negotiation and communication skills, bringing us closer to truly autonomous agents. It highlights the limitations of previous methods and the vast potential for future research in truly understanding and improving AI interactions.