Rico: A Mobile App Dataset for Building Data-Driven Design Applications

By Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Y. Li, Jeffrey Nichols, Ranjitha Kumar. ACM Symposium on User Interface Software and Technology, 2017

724 citations recorded by the source corpus.

Data-driven models help mobile app designers understand best practices and trends, and can be used to make predictions about design performance and support the creation of adaptive UIs. This paper presents Rico, the largest repository of mobile app designs to date, created to support five classes of data-driven applications: design search, UI layout generation, UI code generation, user interaction modeling, and user perception prediction. To create Rico, we built a system that combines crowdsourcing and automation to scalably mine design and interaction data from Android apps at runtime. The Rico dataset contains design data from more than 9.7k Android apps spanning 27 categories. It exposes visual, textual, structural, and interactive design properties of more than 72k unique UI screens. To demonstrate the kinds of applications that Rico enables, we present results from training an autoencoder for UI layout similarity, which supports query- by-example search over UIs.

Subfields: Computer Science Applications, Information Systems, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/775f7845e4df2576762960943294bd28733e2046

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Human-Computer Interaction

Citations in the corpus

Dark Patterns at Scale (2019, background): “than its alternative (e.g., not subscribing). One starting point to detect such interfaces could be to incorporate style and color as features for clustering, or even use the design mining literature [39, 56, 59] to analyze specific types of interfaces (e.g., page headers) in isolation. Finally, res”
Interactive Mobile App Navigation with Uncertain or Under-specified Natural Language Commands (2022, background): “MoTIF subsumes several datasets and topics: web task automation [12, 17, 20, 24, 28], VLN [3, 6, 11], task relevance prediction [10, 13, 27], app design [7, 8, 21]; see Table 1 for comparison.”
SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents (2024, methodology): “To improve element diversity, we also incorporate the automatically generated elements and instructions from RICO (Li et al., 2020a).”
Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, neutral): “…[94], PathVQA (en) [83], Medical PMC-CaseReport (en) [260] Screen2Words (en) [240], WebSight (en) [122], Widget-Caption (en) [136], RICOSCA (en) [55], Seeclick (en) [37], ScreenQA (en) [92], AMEX (en) [22], AITW (en) [198], Odyssey (en) [168], GUI UIBert (en) [12], AndroidControl (en) [135],…”
LayoutGPT: Compositional Visual Planning and Generation with Large Language Models (2023, background): “Image Layout Generation Layout generation has been an important task for automatic graphical design for various scenarios, including indoor scenes [40, 46], document layouts [59, 60, 16], and graphical user interface [8].”
Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image Retrieval (2020, methodology): “This hierarchical information is distinctly defined in the meta-data released with the RICO [8] dataset (an example shown in Figure.”