This study introduces a novel approach to combat hallucinatory phenomena in LVLMs, where models generate texts that don’t align with visual inputs.

• Implements a detect-then-rewrite strategy to correct hallucinations.
• Creates a dataset for training models to recognize and mitigate hallucinations.
• Introduces Hallucination Severity-Aware Direct Preference Optimization for model training.
• The approach aims to enhance the text relevance and accuracy in multi-modal tasks, paving the way for more reliable AI interactions.

Hallucination detection and mitigation refine the outputs of LVLMs, crucially enhancing the reliability of AI outputs in diverse applications.