HTLM: Hyper-Text Pre-Training and Prompting of Language Models

By Armen Aghajanyan, Dmytro Okhonko, Michael Lewis, Mandar Joshi, Hu Xu, Gargi Ghosh, Luke S. Zettlemoyer. 2021

83 citations recorded by the source corpus.

We introduce HTLM, a hyper-text language model trained on a large-scale web crawl. Modeling hyper-text has a number of advantages: (1) it is easily gathered at scale, (2) it provides rich document-level and end-task-adjacent supervision (e.g. class and id attributes often encode document category information), and (3) it allows for new structured prompting that follows the established semantics of HTML (e.g. to do zero-shot summarization by infilling title tags for a webpage that contains the input text). We show that pretraining with a BART-style denoising loss directly on simplified HTML provides highly effective transfer for a wide range of end tasks and supervision levels. HTLM matches or exceeds the performance of comparably sized text-only LMs for zero-shot prompting and fine-tuning for classification benchmarks, while also setting new state-of-the-art performance levels for zero-shot summarization. We also find that hyper-text prompts provide more value to HTLM, in terms of data efficiency, than plain text prompts do for existing LMs, and that HTLM is highly effective at auto-prompting itself, by simply generating the most likely hyper-text formatting for any available training data. We will release all code and models to support future HTLM research.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Prompting & Reasoning

Citations in the corpus

Toolformer: Language Models Can Teach Themselves to Use Tools (2023, methodology): “…LMs with some form of additional textual information during pretraining, including various forms of metadata (Keskar et al., 2019), HTML tags (Aghajanyan et al., 2021), Wikipedia markup (Schick et al., 2022), or related texts obtained from an information retrieval system (Guu et al., 2020;…”
InCoder: A Generative Model for Code Infilling and Synthesis (2022, background): “Recent work addresses this by changing model architectures, inference procedures, and training objectives [2, 59, 67, 3].”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, neutral): “…translation, abstractive summarization, data-to-text generation, and stylistic re-writing, among others) (Raffel et al., 2020; Zhang et al., 2020b; Aghajanyan et al., 2022; Prabhumoye et al., 2018), (2) or open-ended tasks (such as dialogue generation, prompt-based autocompletion, story…”
CM3: A Causal Masked Multimodal Model of the Internet (2022, methodology): “Speciﬁcally, HTLM inserts a probabilistic estimate of the size of the mask as a token post the mask token (e.g., <mask>12 for a probabilistic size of 12).”
Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2021, background): “[1] present structured prompts based on hyper text markup language for more ine-grained web text generation.”
WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents (2022, background): “For example, pre-training with multi-modal data [27, 52], web hypertext [2], or web instruction-action mapping [38] could help agents better understand and leverage rich semantics of webpage content, actions, and instructions.”
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022, background): “Cross-task generalization with PLMs Multi-task learning and transfer learning go beyond task boundaries, view different tasks as related, and have been shown to outperform single-task learning (Aghajanyan et al., 2021a; Vu et al., 2021).”
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023, methodology): “Inspired by Aghajanyan et al. (2022), we removed from the DOM-HTML all the sub-trees contained in a <script>, <style>, <header>, <iframe>, <footer> and <form> tag as well as all the sub-trees associated with a <body>, <div>, <p>, <section>, <table>, <ul>, <ol> or <dl> tag whose textual content was l”
Efficient Training of Language Models to Fill in the Middle (2022, methodology): “Within our modalities of interest, DOBF [Lachaux et al., 2021] trains BERT on code, and HTLM [Aghajanyan et al., 2021] trains BART on HTML data.”
A Review on Language Models as Knowledge Bases (2022, methodology): “However, such a method would be expensive and impractical for modifying a few datapoints, especially for large LMs.”
CCQA: A New Web-Scale Question Answering Dataset for Model Pre-Training (2021, methodology): “, 2020) and the markup-style language modelling HTLM corpus for zero-shot summarization (Aghajanyan et al., 2022b).”
Improving In-Context Few-Shot Learning via Self-Supervised Training (2022, methodology): “…papers have adapted various resources for pretraining models to enhance their performances on few-shot learning, such as pretraining on hypertext (Aghajanyan et al., 2021b), question-infused pre-training (Jia et al., 2021), and self-training (Du et al., 2021; Vu et al., 2021; Wang et al., 2021b).”
LinkBERT: Pretraining Language Models with Document Links (2022, background): “Hyperlinks are also ubiquitous on the web and easily gathered at scale (Aghajanyan et al., 2021).”
Understanding HTML with Large Language Models (2022, background): “The second category pretrains LLMs on a large corpora of HTML text (Aghajanyan et al., 2021).”
Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection (2022, background): “Even lightweight ﬁlters (Aghajanyan et al., 2021; Rae et al., 2021) put more emphasis on features like document length over factuality when determining what makes a document high quality.”
PRIMER: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization (2021, background): “Pre-training on title of HTMLs have been recently shown to be useful for few-shot summarization as well (Aghajanyan et al., 2021).”
Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding (2022, background): “HTLM (Aghajanyan et al., 2022b) and CM3 (Aghajanyan et al., 2022a) are generative models of simpliﬁed HTML to enable zero-shot prompting with text and natural images.”
A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT (2023, background): “HTLM [69] leverages BART denoising objectives for modeling hyper-text language, which contains valuable information regarding document-level structure.”