AppAgent: Multimodal Agents as Smartphone Users

By Chi Zhang, Zhao Yang, Jiaxuan Liu, Yanda Li, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, Gang Yu. International Conference on Human Factors in Computing Systems, 2023

479 citations recorded by the source corpus.

Recent advancements in large language models (LLMs) have led to the creation of intelligent agents capable of performing complex tasks. This paper introduces a novel LLM-based multimodal agent framework designed to operate smartphone applications. Our framework allows the agent to mimic human-like interactions such as tapping and swiping through a simplified action space, eliminating the need for system back-end access and enhancing its versatility across various apps. Central to the agent’s functionality is an innovative in-context learning method, where it either autonomously explores or learns from human demonstrations, creating a knowledge base used to execute complex tasks across diverse applications. We conducted extensive testing with our agent on over 50 tasks spanning 10 applications, ranging from social media to sophisticated image editing tools. Additionally, a user study confirmed the agent’s superior performance and practicality in handling a diverse array of high-level tasks, demonstrating its effectiveness in real-world settings. Our project page is available at https://appagent-official.github.io/.

Subfields: Artificial Intelligence, Social Psychology

View Paper: https://www.semanticscholar.org/paper/61c723e7d91ca308b9eeacab35a2fce7548ef9fc

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models (2024, unknown): “…and further push the boundary of next-level artificial general intelligence, spanning from embodied intelligence (Geng et al., 2023), autonomous driving (Wen et al., 2023; Cao et al., 2023; Yang et al., 2023d) to graphical user interfaces (GUI) agents (He et al., 2024; Yang et al., 2023f).”
A survey on multimodal large language models (2023, unknown): “The image is from [16].”
Promises and challenges of generative artificial intelligence for human learning (2024, unknown): “Diffusion models have shown promise in generating high-quality, realistic data and are used in applications such as image synthesis, text generation and other creative tasks.”
WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models (2024, background): “Imagine you are a robot browsing the web, just like humans.”
AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents (2024, background): “Yet, most existing approaches evaluate agents with proxy metrics, by comparing an agent’s trajectory to a previously collected human demonstration [28, 5, 9, 41, 63, 68, 33, 67, 61].”
VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks (2024, background): “Several recent work have also explored the use of strong VLMs for building visual agents for mobile platforms (Zhan & Zhang, 2023; Chu et al., 2023; Yang et al., 2023b) as well as for the web (Gur et al., 2023; Hong et al., 2023).”
A survey on large language model based autonomous agents (2023, unknown): “5 The applications (left) and evaluation strategies (right) of LLM-based agents. but are highly related to this area.”