MATE: Multi-view Attention for Table Transformer Efficiency

By Julian Martin Eisenschlos, Maharshi Gor, Thomas Müller, William W. Cohen. 2021

108 citations recorded by the source corpus.

This work presents a sparse-attention Transformer architecture for modeling documents that contain large tables. Tables are ubiquitous on the web, and are rich in information. However, more than 20% of relational tables on the web have 20 or more rows (Cafarella et al., 2008), and these large tables present a challenge for current Transformer models, which are typically limited to 512 tokens. Here we propose MATE, a novel Transformer architecture designed to model the structure of web tables. MATE uses sparse attention in a way that allows heads to efficiently attend to either rows or columns in a table. This architecture scales linearly with respect to speed and memory, and can handle documents containing more than 8000 tokens with current accelerators. MATE also has a more appropriate inductive bias for tabular data, and sets a new state-of-the-art for three table reasoning datasets. For HybridQA (Chen et al., 2020), a dataset that involves large documents containing tables, we improve the best prior result by 19 points.

Subfields: Artificial Intelligence, Information Systems, Management Science and Operations Research, Natural Language Processing, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Management Science and Operations Research

In Natural Language Processing

In Model Architectures

In Efficiency & Compression

Citations in the corpus

UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022, background): “Recently, cell-column alignment is modeled by manipulating the attention matrix of transformers (Zhang et al., 2020; Eisenschlos et al., 2021).”