Addressing Long-tail Entities in VQA

The advancement of vision-extended LLMs (VLLMs) has been remarkable, yet their capability in handling long-tail entities lags behind. SnapNTell is introduced as a benchmark with a tailored dataset for entity-centric Visual Question Answering (VQA), testing models on recognizing entities and providing detailed knowledge about them.

In-depth points covered by the research:

• A comprehensive dataset containing categorized entities paired with knowledge-intensive QA pairs.
• Implementation of a retrieval-augmented multimodal LLM for improved performance in VQA tasks.
• SnapNTell’s notable performance improvement by 66.5% on the BELURT score over existing methods.
• Upcoming public access to both the SnapNTell dataset and source code.

This research underscores the need for addressing complex queries involving rare entities in VQA, which may lead to improved factual and detailed responses from AI systems. Emphasizing retrieval-augmentation can serve as a cornerstone for building more accurate and sophisticated multimodal language models.