DocBERT: BERT for Document Classification

By Ashutosh Adhikari, Achyudh Ram, Raphael Tang, Jimmy J. Lin. arXiv.org, 2019

327 citations recorded by the source corpus.

We present, to our knowledge, the first application of BERT to document classification. A few characteristics of the task might lead one to think that BERT is not the most appropriate model: syntactic structures matter less for content categories, documents can often be longer than typical BERT input, and documents often have multiple labels. Nevertheless, we show that a straightforward classification model using BERT is able to achieve the state of the art across four popular datasets. To address the computational expense associated with BERT inference, we distill knowledge from BERT-large to small bidirectional LSTMs, reaching BERT-base parity on multiple datasets using 30x fewer parameters. The primary contribution of our paper is improved baselines that can provide the foundation for future work.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/1a9954d86466a7e4de6f98ddee452ceb50e15d86

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction (2020, unknown): “, texts, with numerous variations [33-39].”
A Survey of the Usages of Deep Learning for Natural Language Processing (2020, methodology): “[197] used BERT [71] to obtain state-ofthe-art classification results on four document data sets.”
Revisiting Transformer-based Models for Long Document Classification (2022, background): “A common practice is to truncate actually long documents to the ﬁrst 512 tokens, which allows the immediate application of these pre-trained models (Adhikari et al., 2019; Chalkidis et al., 2020).”
A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios (2020, background): “This is also displayed in the number of domain-adapted language models (Alsentzer et al., 2019; Huang et al., 2019; Adhikari et al., 2019; Lee and Hsiang, 2020; Jain and Ganesamoorty, 2020, (i.a.)), most notably BioBERT (Lee et al., 2020) that was pre-trained on biomedical PubMED articles and…”
Description Based Text Classification with Reinforcement Learning (2020, methodology): “For multi-label datasets, we follow Adhikari et al. (2019a), in which each label is associated with a binary sigmoid function, and then the document embedding is fed to output the class label.”
BERT Goes to Law School: Quantifying the Competitive Advantage of Access to Large Legal Corpora in Contract Understanding (2019, background): “Recently, numerous studies have shown the value of ﬁne-tuning language models such as ELMo [3] and BERT [4] to achieve state-of-the-art results [5] on domain speciﬁc tasks [6, 7].”
Multi-Stage Pretraining for Low-Resource Domain Adaptation (2020, methodology): “For the TechQA-DR and AskUbuntu-DQD tasks, we follow (Adhikari et al., 2019) and classify the [CLS] token representation at the ﬁnal layer with a binary classiﬁer trained using the binary cross entropy loss; during inference, we rank the documents or questions according to their clas-siﬁcation…”