Language agents powered by Large Language Models (LLMs) are becoming increasingly sophisticated. However, assessing their real-world efficacy requires benchmarks like GroundCocoa which evaluates flight-booking via compositional and conditional reasoning.

• Highlights:

  • GroundCocoa tests LLM agents’ ability to match user preferences with flight options.
  • Even the best performing model, GPT-4 Turbo, achieves only 67% accuracy.
  • The benchmark unveils the disparity in models’ reasoning capabilities.

• Importance: This study underscores the limitations of current LLM agents in performing tasks requiring human-like reasoning. It is crucial for developing more reliable AI systems that can handle everyday tasks with intricacies akin to flight booking.

Read More