World of Bits: An Open-Domain Platform for Web-Based Agents

By Tianlin Shi, Andrej Karpathy, Linxi (Jim) Fan, J. Hernández, Percy Liang. 2017

341 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In LLM Agents & Tool Use

Citations in the corpus

Augmented Language Models: a Survey (2023, background): “other works on web navigation and computer-control, most of them assume the typical human interface, that takes as input images of a computer screen and output keyboard commands in order to solve digital tasks (Shi et al., 2017; Gur et al., 2019; 2021; Toyama et al., 2021; Humphreys et al., 2022; Gu”
WebGPT: Browser-assisted question-answering with human feedback (2021, methodology): “They train their system in two alternative ways: behavior cloning (BC) on synthetically-generated sequences and RL. Finally, there is another body of work that uses BC and RL to control web browsers, for automating other tasks besides question-answering [Shi et al., 2017, Gur et al., 2018].”
WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents (2022, background): “[43]), we believe that it provides a very low-level non-semantic action space.”
Building Generalizable Agents with a Realistic and Rich 3D Environment (2018, neutral): “, 2013) • OpenAI Universe (Shi et al., 2017) • • • Malmo (Johnson et al.”
WebArena: A Realistic Web Environment for Building Autonomous Agents (2023, methodology): “As a result, the functionality of many environments is a limited version of their real-world counterparts, leading to a lack of task diversity (Shi et al., 2017; Anderson et al., 2018; Gordon et al., 2018; Misra et al., 2016; Shridhar et al., 2020; 2021; Yao et al., 2022a).”
Cognitive Architectures for Language Agents (2023, background): “…2022a; Liu et al., 2023d), APIs (Schick et al., 2023; Yao et al., 2022b; Parisi et al., 2022; Tang et al., 2023b), and websites (Shi et al., 2017; Nakano et al., 2021; Yao et al., 2022a; Zhou et al., 2023b; Gur et al., 2023; Deng et al., 2023) as well as general code execution (Yang et al., 2023;…”
FLIN: A Flexible Natural Language Interface for Web Navigation (2020, neutral): “Work on NL-guided web navigation includes learning tasks from demonstrations (Allen et al., 2007), building reinforcement learning agents (Shi et al., 2017b; Liu et al., 2018b), semantic parsing using seq2seq models to map NL commands into web APIs (Su et al., 2017, 2018) and generating task ﬂows…”
VIMA: General Robot Manipulation with Multimodal Prompts (2022, background): “, 2023), powered by a collection of learning environments (Abramson et al., 2020; Shridhar et al., 2020; Savva et al., 2019; Puig et al., 2018; Team et al., 2021; Toyama et al., 2021; Shi et al., 2017).”
A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis (2023, methodology): “, 2022), have a capability of manipulating the web environments (Shi et al., 2017) with great HTML comprehension (Furuta et al.”
Interactive Mobile App Navigation with Uncertain or Under-specified Natural Language Commands (2022, background): “This results in natu-ral language queries that mimic real users, unlike automatically generated tasks from prior work, which all hold the same templated form ( e.g ., MiniWoB [28] creates versions of ‘Click on x ’ by varying x ).”
Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration (2018, methodology): “Since we have removed partial rewards, success rate is a linear scaling of the average reward, and is equivalent to the deﬁnition of success rate in Shi et al. (2017).”
AgentBench: Evaluating LLMs as Agents (2023, background): “More recently, attempts on embodied agents [63; 35; 2] have employed complicated multi-modal simulators based on games [41; 22], GUI [69; 81], and indoor scenes [68; 74].”
AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation (2023, neutral): “We demonstrate how to use AutoGen to build a working system for handling such scenarios with the MiniWoB++ benchmark (Shi et al., 2017).”
GPT-4V(ision) is a Generalist Web Agent, if Grounded (2024, methodology): “Pix2Act (Shaw et al., 2023) leverages Pix2Struct (Lee et al., 2022) to parse screenshot images into simplified HTML to complete GUI-based tasks (Shaw et al., 2023; Liu et al., 2018; Shi et al., 2017; Mazumder & Riva, 2020; Yao et al., 2022).”
A data-driven approach for learning to control computers (2022, methodology): “Shown are previously published MiniWob++ results (Gur et al., 2018; Jia et al., 2019; Liu et al., 2018; Shi et al., 2017), which each only report performance for a subset of tasks (for these results the performance on non-reported tasks is set to zero, all scores used are given in Appendix Table 3).”
Mind2Web: Towards a Generalist Agent for the Web (2023, background): “Yet, previous research has been limited by the types of tasks and websites it can handle, either confined to simplified simulation environments [22, 31, 40], or limited to a narrow set of domains and websites [39, 40].”
Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (2024, neutral): “…general multimedia understanding and generation capabilities of (multimodal) LLMs empower GUI agents to generalize beyond simple simulated settings (Shi et al., 2017; Humphreys et al., 2022) to diverse and complex real-world environments, including the web (Deng et al., 2023; Zhou et al., 2024;…”
DOM-Q-NET: Grounded RL on Structured Language (2019, background): “Liu et al. (2018) achieved state-of-the-art performance with very few expert demonstrations in the MiniWoB (Shi et al., 2017) benchmark tasks, but their exploration policy requires constrained action sets, hand-crafted with expert knowledge in HTML.”
SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents (2024, methodology): “Due to the variance in evaluation task sets among different methods (Liu et al., 2018; Furuta et al., 2023; Shaw et al., 2023), for fairness, we report performance in two groups based on the overlapping MiniWob tasks with their approaches.”