OmniDrive introduces a groundbreaking approach in the integration of Large Language Models (LLMs) with 3D perception technologies for autonomous driving. The framework encapsulates a novel 3D MLLM architecture that enhances both the perception and planning stages of autonomous vehicles. Key highlights include:

• A unique 3D query-based multimodal LLM (MLLM) architecture that lifts and compresses visual data into 3D.
• OmniDrive-nuScenes, a new visual question-answering dataset tailored for evaluating true 3D situational awareness.
• Extensive validation demonstrates the effectiveness of the architecture in complex 3D scenes.

Significance: This research not only advances the field of autonomous driving but also sets a new standard for the implementation of AI technologies in real-world applications. The OmniDrive framework could serve as a blueprint for future developments in vehicle autonomy, emphasizing the importance of 3D cognitive capabilities in enhancing situational awareness and response accuracy.