This research paper presents the novel task of multimodal puzzle solving, with the creation of AlgoPuzzleVQA—a new dataset aimed at evaluating the capabilities of multimodal language models such as GPT4V and Gemini in solving intricate puzzles that require both visual understanding and algorithmic reasoning.

• The dataset covers various mathematical and algorithmic topics, including boolean logic, combinatorics, and graph theory.
• Generated from human-authored code, it offers puzzles with exact solutions that can be algorithmically determined, allowing for scalability in reasoning complexity.
• Findings suggest that large language models perform poorly on these puzzle-solving tasks, with near-random performance in many instances.
• The study underscores the difficulty of integrating visual, linguistic, and algorithmic knowledge to solve complex reasoning problems, marking a substantial gap in current AI capabilities.

The implications of this study are profound, highlighting the current limitations of AI in performing tasks that require multimodal inputs and complex reasoning. It sets the stage for future advancements in the field, emphasizing the need for AI systems capable of more sophisticated integrations of different forms of knowledge.