The paper OlympiadBench introduces a bilingual multimodal scientific benchmark taken from Olympiad-level competitions, challenging top-tier AI models like GPT-4V. The benchmark covers 8,952 problems, pushing the envelope of AI capabilities.

• OlympiadBench: a new standard for evaluating AI expertise.
• Challenging tasks emphasize reasoning and knowledge.
• GPT-4V scores reflect the difficulty of translating human expertise to AI.
• The need to address issues such as hallucinations and logical fallacies is highlighted.

This paper emphasizes the gap between AI and human expertise in areas requiring deep scientific understanding and critical reasoning. It showcases the importance of benchmarks that go beyond traditional tasks, pushing AI towards genuine artificial general intelligence.