Abstract

User interface understanding with vision-language models has received much attention due to its potential for enabling next-generation software automation. However, existing UI datasets either only provide large-scale context-free element annotations or more contextualized functional descriptions for elements at a much smaller scale. In this work, we propose the AutoGUI pipeline for automatically annotating UI elements with detailed functionality descriptions at scale. Specifically, we leverage large language models (LLMs) to infer element functionality by comparing the UI source code changes before and after simulated interactions with specific UI elements. We construct an AutoGUI-627k dataset using the proposed pipeline, featuring multi-resolution, multi-device screenshots, diverse data domains, and detailed functionality annotations. Furthermore, we propose a spatially-aware instruction tuning approach for enhancing the grounding ability of UI vision-language models (UI-VLMs). Extensive experimental results demonstrate the superiority of our AutoGUI-627k dataset compared to existing web pre-training methods and the effectiveness of the proposed spatially-aware instruction tuning approach.

Overview of AutoGUI Dataset

AutoGUI contains 627k UI grounding/captioning tasks involving 861 domains from Common Crawl. Different from exsiting UI datasets, AutoGUI provides abundant UI functionality annotations without human experts and focuses on functionality grounding/captioning tasks.

Task 1: Functionality Grounding: Given a user's instruction that describes UI functionalities, models are prompted to output the 2D coordinates of the element targeted by the instruction. Note that the coordinates are normalized withing [0,100)
Example:

Task 2: Functionality Captioning: Given a user's instruction that refers to a certain location in the format of a point, models are prompted to describe the functionality of the pointed element considering the UI context.
Example:

Every sample in the training set contains the following information:

• Functionality that describes the contextual functionality of an element.
• Target Element that provides the attributes of the element associated with the functionality label.
• Element Position: the original bounding box coordinates [left,top,right,bottom] as well as the center point (X,Y)normalized within the range 0-100 are provided.
• Element Text: The displayed text of alt-text of the element.
• Device Type: Whether the task is collected in a view of a web browser or mobile phone.
• UI Screenshot that records the UI image on which the element is displayed: