Sikuli: using GUI screenshots for search and automation

By Tom Yeh, Tsung-Hsiang Chang, Rob Miller. ACM Symposium on User Interface Software and Technology, 2009

446 citations recorded by the source corpus.

We present Sikuli, a visual approach to search and automation of graphical user interfaces using screenshots. Sikuli allows users to take a screenshot of a GUI element (such as a toolbar button, icon, or dialog box) and query a help system using the screenshot instead of the element's name. Sikuli also provides a visual scripting API for automating GUI interactions, using screenshot patterns to direct mouse and keyboard events. We report a web-based user study showing that searching by screenshot is easy to learn and faster to specify than keywords. We also demonstrate several automation tasks suitable for visual scripting, such as map navigation and bus tracking, and show how visual scripting can improve interactive help systems previously proposed in the literature.

Subfields: Sociology and Political Science, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/203091a855582e07ea0ae0b8382d7964c3ed46fd

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Sociology and Political Science

Percy Liang

In Human-Computer Interaction

Citations in the corpus

PLUS: open-source toolkit for ultrasound-guided intervention systems (2014, unknown): “[36] T. Yeh, T.-H. Chang, and R. C. Miller, “Sikuli: using gui screenshots for search and automation,” in Proc.”
Mapping natural language commands to web elements (2018, background): “While most scripts reference elements with logical selectors (e.g., CSS and XPath), there have been several alternatives such as images (Yeh et al., 2009) and simple natural language utterances (Soh, 2017).”
Combining Design and Performance in a Data Visualization Management System (2017, methodology): “One way to solve this problem is to let users compose their own interfaces, with tools such as Mac Automator, Scratch [43] or Sikuli [62].”
Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration (2018, background): “First, simple programs may be speciﬁed by the user (Yeh et al., 2009) or may be inferred from demonstrations (Allen et al., 2007).”