Understanding Multi-modal Large Language Models in Visual Math Problems

Recent advancements in Multi-modal Large Language Models (MLLMs) have shown impressive performance in visual tasks. However, their effectiveness in visual math problem-solving has been underexplored. The new benchmark MathVerse aims to address this gap by providing 2,612 high-quality math problems with diagrams, covering multiple subjects. These problems are transformed into six versions with varying multi-modality information, totaling 15K unique test samples.

• The benchmark evaluates whether MLLMs truly understand visual diagrams for mathematical reasoning.
• A Chain-of-Thought (CoT) evaluation strategy extracts critical reasoning steps using GPT-4(V) for error analysis.
• MathVerse can highlight intermediate reasoning quality, offering insights for future MLLM development.
• Visit the project page for more details.

This study underscores the importance of thorough evaluation frameworks to advance our comprehension of MLLM capabilities. By dissecting the reasoning process, MathVerse not only tests MLLM’s problem-solving skills but also paves the way for models that can ‘see’ and ‘understand’ visual elements more profoundly.