SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents

By Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, Zhiyong Wu. Annual Meeting of the Association for Computational Linguistics, 2024

561 citations recorded by the source corpus.

Graphical User Interface (GUI) agents are designed to automate complex tasks on digital devices, such as smartphones and desktops. Most existing GUI agents interact with the environment through extracted structured data, which can be notably lengthy (e.g., HTML) and occasionally inaccessible (e.g., on desktops). To alleviate this issue, we propose a novel visual GUI agent -- SeeClick, which only relies on screenshots for task automation. In our preliminary study, we have discovered a key challenge in developing visual GUI agents: GUI grounding -- the capacity to accurately locate screen elements based on instructions. To tackle this challenge, we propose to enhance SeeClick with GUI grounding pre-training and devise a method to automate the curation of GUI grounding data. Along with the efforts above, we have also created ScreenSpot, the first realistic GUI grounding benchmark that encompasses mobile, desktop, and web environments. After pre-training, SeeClick demonstrates significant improvement in ScreenSpot over various baselines. Moreover, comprehensive evaluations on three widely used benchmarks consistently support our finding that advancements in GUI grounding directly correlate with enhanced performance in downstream GUI agent tasks. The model, data and code are available at https://github.com/njucckevin/SeeClick.

Subfields: Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/f9b39a6a7e40986b46f7796f3a805d70d7e3931a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Zhenyu Wu
2. Kanzhi Cheng
3. Boyu Gou
4. Boyuan Zheng
5. Qiushi Sun
6. Fangzhi Xu
7. Huan Sun
8. Yian Wang

Citations in the corpus

• AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents (2024, unknown): “Recent work explores agents that reflect on system state [46, 65, 36] by leveraging exploration, self-evaluation, and retry-capabilities enabling continual learning and adaptation [27, 63, 39, 58].”
• Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (2024, unknown): “More details are in Appendix F.”
• GPT-4V(ision) is a Generalist Web Agent, if Grounded (2024, background): “LMMs can generate target element coordinates in the image via training on datasets augmented with object coordinates, especially for open-sourced models (Hong et al., 2023; Cheng et al., 2024; You et al., 2023).”
• InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency (2025, unknown): “MMLU-Pro [61], GAOKAO [177], IFEval [185]; (4) Agentic Tasks : SGP-Bench [102], ScreenSpot [16], ScreenSpot-v2 [150], OSWorld-G [156], VSI-Bench [161], ERQA [121], SpaCE-10 [38].”
• OS-ATLAS: A Foundation Action Model for Generalist GUI Agents (2024, methodology): “Recent efforts have begun to enable agents to interact with operating systems via programs (Sun et al., 2024) or API calls (Wu et al., 2024; Zhang et al., 2024a).”
• Qwen3-VL Technical Report (2025, unknown): “We evaluate UI perception with GUI-grounding tasks (ScreenSpot (Cheng et al., 2024), ScreenSpot Pro (Li et al., 2025b), OSWorldG(Xie et al., 2025a)) and assess decision-making abilities through online environment evaluations (AndroidWorld (Rawles et al., 2024), OSWorld (Xie et al., 2025c;b)).”
• Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, unknown): “…[83], Medical PMC-CaseReport (en) [260] Screen2Words (en) [240], WebSight (en) [122], Widget-Caption (en) [136], RICOSCA (en) [55], Seeclick (en) [37], ScreenQA (en) [92], AMEX (en) [22], AITW (en) [198], Odyssey (en) [168], GUI UIBert (en) [12], AndroidControl (en) [135], Mind2Web (en) [57],…”
• SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models (2024, unknown): “- ScreenSpot (Cheng et al., 2024): A benchmark across various GUI platforms and designed to assess MLLM’s capability to localize elements based on the human’s instructions.”
• Leveraging Vision-Language Models for Visual Grounding and Analysis of Automotive UI (2025, unknown): “To determine if our fine-tuning resulted in overfitting to the automotive UI domain, we applied ELAM-7B to the ScreenSpot dataset [3] comparing it to SeeClick [3], ShowUI [18], OS-Atlas-Base-7B [33], Molmo-7B-D-0924/Molmo-72B-0924 [4], and Qwen2-VL-based UGround-V1-7B [10] in Table 4.”