Summary

• MANTIS introduces a novel approach to enhance the performance of large multimodal models (LMMs) capable of handling multi-image tasks.

• By focusing on instruction tuning rather than extensive pre-training, MANTIS achieves top performance using fewer resources.

• It includes the development of Mantis-Instruct, a new dataset comprising 721K instances from 14 multi-image datasets, aimed at improving specific skills like co-reference, reasoning, comparing, and temporal understanding.

  Detailed Findings

• The new training strategy using Mantis-Instruct and existing single-image datasets enhances LMMs capabilities substantially, showing superior performance on both multi-image and single-image benchmarks compared to the baseline models.

• Its ability to simultaneously manage multiple images and conduct complex visual reasoning tasks makes it uniquely positioned for advancements in AI visual understanding.

  Author’s Comments

• ‘This work not only pushes forward the efficiency and effectiveness of multi-image LLMs but also highlights the importance of instruction-based training in achieving superior outcomes.’

• ‘Mantis demonstrates that strategic instruction tuning can outperform traditional pre-training methods, setting new standards for LMM development.’

  Opinion

  MANTIS represents a significant leap forward in the practical application of large visual language models, particularly in multi-image tasks. Its innovative use of academic resources for instruction tuning showcases a sustainable and scalable model for LLM development that could inspire future research in various fields of artificial intelligence, especially in scenarios requiring sophisticated visual comprehension.