VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks

By Jing Yu Koh, Robert Lo, L. Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, Daniel Fried. Annual Meeting of the Association for Computational Linguistics, 2024

628 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/f554b22d2ccf786a6d61d5858f43024ba9115e15

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

LongVILA: Scaling Long-Context Visual Language Models for Long Videos (2024, background): “…more modalities can take more flexible input signals so that people can interact with the model in more diverse manners, e.g., GPT-4o-like multi-modal chatbot, multi-modal web agent (Koh et al., 2024), and real-world robotics foundation model (Brohan et al., 2022; 2023; Padalkar et al., 2023).”
WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models (2024, background): “Figure 23 illustrates a snippet of WebVoyager navigating on Google Flights and a Visual Grounding Issue appears.”
MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark (2024, background): “This ability is crucial for tasks ranging from interpreting scientific diagrams (Li et al., 2024d) to navigating graphical user interfaces (Liu et al., 2024b; Zheng et al., 2024; Koh et al., 2024).”
AppAgent: Multimodal Agents as Smartphone Users (2023, neutral): “Recent systems like Mind2Web [8], WebArena [53], WorkArena [9], and VisualWebArena [18] have leveraged LLMs to operate and interact with web pages, while others, such as OSWorld [41], extend similar capabilities to desktop PCs.”
AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents (2024, methodology): “More recent work tends to leverage off-the-shelf foundational models [50, 51, 52] with in-context learning (ICL) and fine-tuning applied to mobile [41, 18, 56, 61, 68, 4], desktop web [70, 9, 73, 24], and desktop OS [58, 66, 59].”
Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (2024, neutral): “2 Only with the recent advances in multimodal LLMs (MLLMs) does visual perception become broadly viable, but text-based representations are still used jointly (Zheng et al., 2024; Koh et al., 2024; Zhang et al., 2024a).”
Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models (2024, methodology): “Researchers have applied this idea to software engineering [20, 61, 15], robotics [34], vision [29, 57], and GUI navigation [54, 23, 53].”