Advances in AI have expanded the potential applications of LLMs in complex environments like autonomous driving. This research introduces a novel 3D Multimodal Large Language Model (3D MLLM) that processes dynamic objects and map elements, propelling the capabilities of autonomous agents. Key findings of the study:

• Development of comprehensive visual reasoning and planning capabilities for real-world driving scenarios
• Introduction of a new visual question-answering dataset to test models
• Extensive testing showing effective situational awareness and planning OmniDrive’s approach highlights the significance of aligning computational capability with realistic driving tasks, suggesting a forward path for integrating advanced reasoning models in vehicles.