pix2code: Generating Code from a Graphical User Interface Screenshot
By Tony Beltramelli. 2017
370 citations recorded by the source corpus.
Transforming a graphical user interface screenshot created by a designer into computer code is a typical task conducted by a developer in order to build customized software, websites, and mobile applications. In this paper, we show that deep learning methods can be leveraged to train a model end-to-end to automatically reverse engineer user interfaces and generate code from a single input image with over 77% of accuracy for three different platforms (i.e. iOS, Android and web-based technologies).
Subfields: Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Graham Neubig
- Kevin Ellis
- Armando Solar-Lezama
- Frank F. Xu
- Daniel Ritchie
- Bogdan Vasilescu
- Josh Tenenbaum
- Luc Van Gool
Citations in the corpus
- Neural Program Synthesis with Priority Queue Training (2018, neutral): “…2016; Khanh Dam et al., 2016; Neelakantan et al., 2017; Riedel et al., 2016; Schaechtle et al., 2016; Zaremba et al., 2016; Barone & Sennrich, 2017; Beltramelli, 2017; Cai et al., 2017; Devlin et al., 2017; Hu et al., 2017; Guu et al., 2017; Johnson et al.; Liang et al., 2017; Murali et al., 2017;…”
- Learning to Infer Graphics Programs from Hand-Drawn Images (2017, background): “IM2LATEX [19] and pix2code [20] are recent works that derender LATEX equations and GUIs, respectively, both recovering a markup-like representation.”
- In-IDE Code Generation from Natural Language: Promise and Challenges (2021, positive): “In specific domains such as user interface design and prototyping, recent advances in deep learning models also enable developers to sketch the user interface visually and then automatically generates user interface code with the sketch [13], or using existing screenshots [84].”