Understanding HTML with Large Language Models
By Izzeddin Gur, Ofir Nachum, Yingjie Miao, Mustafa Safdari, Austin Huang, Aakanksha Chowdhery, Sharan Narang, Noah Fiedel, Aleksandra Faust. 2022
95 citations recorded by the source corpus.
Large language models (LLMs) have shown exceptional performance on a variety of natural language tasks. Yet, their capabilities for HTML understanding -- i.e., parsing the raw HTML of a webpage, with applications to automation of web-based tasks, crawling, and browser-assisted retrieval -- have not been fully explored. We contribute HTML understanding models (fine-tuned LLMs) and an in-depth analysis of their capabilities under three tasks: (i) Semantic Classification of HTML elements, (ii) Description Generation for HTML inputs, and (iii) Autonomous Web Navigation of HTML pages. While previous work has developed dedicated architectures and training procedures for HTML understanding, we show that LLMs pretrained on standard natural language corpora transfer remarkably well to HTML understanding tasks. For instance, fine-tuned LLMs are 12% more accurate at semantic classification compared to models trained exclusively on the task dataset. Moreover, when fine-tuned on data from the MiniWoB benchmark, LLMs successfully complete 50% more tasks using 192x less data compared to the previous best supervised model. Out of the LLMs we evaluate, we show evidence that T5-based models are ideal due to their bidirectional encoder-decoder architecture. To promote further research on LLMs for HTML understanding, we create and open-source a large-scale HTML dataset distilled and auto-labeled from CommonCrawl.
Subfields: Artificial Intelligence, Information Systems, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Yejin Choi
- Jianfeng Gao
- Hinrich Schütze
- Percy Liang
- Yoshua Bengio
- Timo Schick
In Information Systems
- Huan Sun
- Stephen Marcus McAleer
- Pierre Baldi
- Xiang Deng
- Yann LeCun
- Asli Celikyilmaz
- Yu Su
- Geunwoo Kim
In Natural Language Processing
- Timo Schick
- Asli Celikyilmaz
- Thomas Scialom
- Hinrich Schütze
- Edouard Grave
- Maria Lomeli
- Yejin Choi
- Jane Dwivedi-Yu
Citations in the corpus
- Augmented Language Models: a Survey (2023, background): “While there are other works on web navigation and computer-control, most of them assume the typical human interface, that takes as input images of a computer screen and output keyboard commands in order to solve digital tasks (Shi et al., 2017; Gur et al., 2019; 2021; Toyama et al., 2021; Humphreys ”
- AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents (2024, methodology): “For MobileMiniWoB++, similarly to recent work, we evaluate on a subset of 62 tasks [72, 22, 15].”
- Language Models can Solve Computer Tasks (2023, methodology): “Recent work [18] has suggested the use of large language models (LLMs) to comprehend HTML code, with a few-shot in-context approach showing promising results without extensive RL exploration.”
- A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis (2023, methodology): “method [24] while outperforming naive local-global attention models [22] and its instruction-finetuned variants [12, 19].”
- Mind2Web: Towards a Generalist Agent for the Web (2023, methodology): “The first directly uses the candidate generation model (DeBERTa) for element selection, which is similar to existing work [10, 31] that combines an encoder with classification heads.”