Revisiting Transformer-based Models for Long Document Classification

By Xiang Dai, Ilias Chalkidis, Sune Darkner, Desmond Elliott. 2022

102 citations recorded by the source corpus.

The recent literature in text classification is biased towards short text sequences (e.g., sentences or paragraphs). In real-world applications, multi-page multi-paragraph documents are common and they cannot be efficiently encoded by vanilla Transformer-based models. We compare different Transformer-based Long Document Classification (TrLDC) approaches that aim to mitigate the computational overhead of vanilla transformers to encode much longer text, namely sparse attention and hierarchical encoding methods. We examine several aspects of sparse attention (e.g., size of local attention window, use of global attention) and hierarchical (e.g., document splitting strategy) transformers on four document classification datasets covering different domains. We observe a clear benefit from being able to process longer text, and, based on our results, we derive practical advice of applying Transformer-based models on long document classification tasks.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Christopher R'e
  2. Tri Dao
  3. Atri Rudra
  4. Stefano Ermon
  5. Daniel Y. Fu
  6. Sergey Levine
  7. Pieter Abbeel
  8. Yang Song

In Natural Language Processing

  1. Atri Rudra
  2. Christopher R'e
  3. Tri Dao
  4. Stefano Ermon
  5. Alexander J. Ratner

In Model Architectures

  1. Tri Dao
  2. Christopher R'e
  3. Daniel Y. Fu
  4. Stefano Ermon
  5. Atri Rudra
  6. Albert Gu
  7. Beidi Chen
  8. Karan Goel

Citations in the corpus