Mobile user interfaces (UIs) are complex and vary in design, posing challenges for general-domain multimodal large language models (MLLMs) in effective comprehension and interaction. The paper on Ferret-UI showcases an MLLM tailored specifically for mobile UI screens, incorporating ‘any resolution’ magnification for finer details and robust visual features, while also splitting screens into sub-images for LLM encoding. Ferret-UI was trained on a vast array of UI tasks and establishes a comprehensive benchmark for model evaluation, where it outperforms most open-source UI MLLMs and even GPT-4V on elementary tasks. In summary:

• Ferret-UI addresses the unique challenges of mobile UI screens in semantic tasks.
• The model employs a divide-and-encode strategy for enhanced visual understanding.
• It has been trained on extensive UI tasks for precise grounding and referring.
• Ferret-UI sets a new standard for mobile UI understanding models, outshining its predecessors in elementary tasks.

Ferret-UI’s specialized approach to understanding mobile UI screens is a pivotal advancement, suggesting pathways for further enhancement of AI models in device interaction. As mobile UI design continues to evolve, this kind of tailored AI could play a vital role in developing more intuitive and responsive human-computer interactions.