Summary of the Paper

The paper addresses two fundamental issues in evaluating LVLMs. Firstly, the overestimation of multi-modal gains due to the ability of models to infer answers without visual input. Secondly, unintentional data leakage that has allowed LVLMs to memorize certain visual-necessary questions. To combat these issues, the researchers propose MMStar, a new benchmark to better evaluate multi-modal capacities of LVLMs.

*Key Takeaways:

• Over-reliance on text-only capabilities can misguide the progression of LVLM studies.
• MMStar introduces thorough human selection processes to ensure minimal data leakage and high dependency on visual content.
• Evaluates 16 leading LVLMs and provides new metrics for measuring data leakage and true multi-modal gains.

Why is this important? This paper pinpoints the pitfalls in LVLM evaluations, leading us to re-think the benchmarks we use. Understanding the true capabilities of multi-modal models is crucial for advancing AI technologies in image and language processing fields. The proposed MMStar benchmark and the insights from this study potentially pave the way for creating more reliable and effective LVLMs.

Read the full paper here: Are We on the Right Way for Evaluating Large Vision-Language Models?