The RoboEXP system presents a transformative approach to robotic manipulation, combining Large Multimodal Models with an explicit memory design. By actively interacting with unknown environments, robots can produce action-conditioned scene graphs that detail both geometry and semantics. The robustness of RoboEXP is demonstrated in various real-world settings, proving its value in handling rigid, articulated, nested, and deformable objects within complex tasks.

• The introduction of RoboEXP system for robotic interaction
• Generation of action-conditioned scene graphs for environmental understanding
• Incorporation of LMMs for improved capabilities
• Successful application in complex real-world manipulation tasks

This work serves as a testament to the capabilities of intelligent systems in comprehending and interacting with their environment in dynamic and nuanced ways. Read More