This research critiques the current methods of evaluating Large Vision-Language Models (LVLMs) and presents MMStar, a curated benchmark to thoroughly assess multi-modal capacities. The principal concerns with existing benchmarks are twofold: the superfluity of visual content for many samples and unintentional data leakage during training. The study highlights:

• How LVLMs and LLMs inadvertently memorize parts of training data, leading to no visual requirement for making certain inferences.
• The creation of MMStar, comprising 1,500 samples with minimal data leakage that truly necessitate visual information for the resolution.
• Introduction of metrics to quantify data leakage and evaluate actual performance gains in multi-modal training.
• Extensive evaluation of leading LVLMs to appraise their multi-modal abilities against the new benchmark and metrics.

The significance of this paper stems from its scrutiny of current LVLM evaluation methods. By ensuring purer and more stringent benchmarks, the paper provides a clearer pathway for discerning true multi-modal proficiency, which is essential for applications like interactive robotics and enhanced content analysis. Further study could drive advancements in model training and evaluation to develop more finely-tuned and effective multi-modal AI systems. Read the full paper.