CogAgent: A Visual Language Model for GUI Agents

By Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding and 1 more. Computer Vision and Pattern Recognition, 2023

804 citations recorded by the source corpus.

People are spending an enormous amount of time on dig-ital devices through graphical user interfaces (GUIs), e.g., computer or smartphone screens. Large language models (LLMs) such as ChatGPT can assist people in tasks like writing emails, but struggle to understand and interact with GUIs, thus limiting their potential to increase automation levels. In this paper, we introduce CogAgent, an 18-billion-parameter visual language model (VLM) specializing in GUI understanding and navigation. By utilizing both low-resolution and high-resolution image encoders, CogA-gent supports input at a resolution of1120 × 1120, enabling it to recognize tiny page elements and text. As a general-ist visual language model, CogAgent achieves the state of the art on five text-rich and four general VQA benchmarks, including VQAv2, OK- VQA, Text- Vqa, St- Vqa, ChartQA, infoVQA, DocVQA, MM-Vet, and POPE. CogAgent, using only screenshots as input, outperforms LLM-based methods that consume extracted HTML text on both PC and Android GUI navigation tasks-Mind2Web and AITW, ad-vancing the state of the art. The model and codes are available at https://github.com/THUDM/CogVLM.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/e50583008fe4ec049e42fdc01727ce98f6d86a35

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jie Tang
2. Graham Neubig
3. Hongsheng Li
4. Karthik Narasimhan
5. Ming Ding
6. Yu Su
7. Yuxiao Dong
8. Huan Sun

In Computer Vision and Pattern Recognition

1. Hongsheng Li
2. Ming Ding
3. Jie Tang
4. Yu Liu
5. Graham Neubig
6. Xiaogang Wang
7. Jing Yu Koh
8. Guanglu Song

Citations in the corpus

• Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (2024, neutral): “H YBRID Following prior work (Hong et al., 2024; Cheng et al., 2024), we download and randomly sample from the latest Common Crawl 8 .”
• Cognitive Architectures for Language Agents (2023, background): “For example, Wikipedia is not controllable : it is an external environment that may be unexpectedly modified by other users.”
• How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites (2024, neutral): “Currently, there are two common technical routes: one involves designing a dual-branch image encoder [32, 72, 73, 116], and the other involves dividing a high-resolution image into many low-resolution tiles [33, 46, 52, 53, 60, 64, 121, 122].”
• SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents (2024, methodology): “Our SeeClick achieved the best average performances across three GUI platforms and two types of elements, even with fewer model parameters and GUI-related training data than CogAgent.”
• GPT-4V(ision) is a Generalist Web Agent, if Grounded (2024, methodology): “5 (Liu et al., 2023a;c), and CogA-gent (Hong et al., 2023).”
• VideoAgent: Long-form Video Understanding with Large Language Model as Agent (2024, neutral): “Conversely, visual language models (VLMs) struggle to model lengthy visual inputs [9,17,18,21,23].”
• MM-LLMs: Recent Advances in MultiModal Large Language Models (2024, neutral): “5 (Lv et al., 2023), InternLM-XComposer (Zhang et al., 2023f), JAM (Aiello et al.)”
• LVBench: An Extreme Long Video Understanding Benchmark (2024, positive): “Building upon the achievements in LLMs, the field has shifted towards MLLMs to enhance multi-modal understanding capabilities [2, 11, 26, 32, 38].”
• AppAgent: Multimodal Agents as Smartphone Users (2023, neutral): “For example, CogAgent [16] uses millions of instances from high-resolution GUI data and VQA datasets.”
• VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks (2024, background): “Several recent work have also explored the use of strong VLMs for building visual agents for mobile platforms (Zhan & Zhang, 2023; Chu et al., 2023; Yang et al., 2023b) as well as for the web (Gur et al., 2023; Hong et al., 2023).”
• AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents (2024, background): “More recent work tends to leverage off-the-shelf foundational models [50, 51, 52] with in-context learning (ICL) and fine-tuning applied to mobile [41, 18, 56, 61, 68, 4], desktop web [70, 9, 73, 24], and desktop OS [58, 66, 59].”
• GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning (2025, neutral): “From solving scientific problems [67; 32; 34] to developing autonomous agents [21; 62; 43], the demands on VLMs have far surpassed simple visual content perception [31], with an increasing emphasis on advanced reasoning abilities.”
• OS-ATLAS: A Foundation Action Model for Generalist GUI Agents (2024, methodology): “Consequently, research shifts toward GUI-based agents that interact with digital devices through human-like mouse and keyboard actions (Cheng et al., 2024; Hong et al., 2024b; Zheng et al., 2024a).”
• Group-in-Group Policy Optimization for LLM Agent Training (2025, neutral): “The use of large language models (LLMs) as autonomous agents has expanded rapidly across domains such as program generation [22], smart device operation [23, 24, 25, 26], interactive gameplay [10], and robot behavior control [27].”
• Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning (2024, background): “On the other hand, studies have also explored prompting models [16, 73, 74, 44, 77] to understand complex visual scenes and tasks, focusing on the details of prompting techniques in MLLMs.”
• A survey on multimodal large language models (2023, neutral): “The image is from [16].”