Fine-tune BERT with Sparse Self-Attention Mechanism

By Baiyun Cui, Yingming Li, Ming Chen, Zhongfei Zhang. 2019

73 citations recorded by the source corpus.

In this paper, we develop a novel Sparse Self-Attention Fine-tuning model (referred as SSAF) which integrates sparsity into self-attention mechanism to enhance the fine-tuning performance of BERT. In particular, sparsity is introduced into the self-attention by replacing softmax function with a controllable sparse transformation when fine-tuning with BERT. It enables us to learn a structurally sparse attention distribution, which leads to a more interpretable representation for the whole input. The proposed model is evaluated on sentiment analysis, question answering, and natural language inference tasks. The extensive experimental results across multiple datasets demonstrate its effectiveness and superiority to the baseline methods.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Qun Liu
2. Xin Jiang
3. Lifeng Shang
4. Lu Hou
5. Sanjiv Kumar
6. Sashank J. Reddi
7. Ankit Singh Rawat
8. Zhiqi Huang

Citations in the corpus

• DynaBERT: Dynamic BERT with Adaptive Width and Depth (2020, background): “There have been some attempts to compress and accelerate inference of the Transformer-based models using low-rank approximation [15, 12], weight-sharing [4, 12], knowledge distillation [21, 23, 10, 28], quantization [1, 33, 22, 8] and pruning [16, 17, 3, 25].”
• Seeking Patterns, Not just Memorizing Procedures: Contrastive Learning for Solving Math Word Problems (2021, background): “, 2019a) has provided informative contextual representations for text understanding, and fine-tuning techniques (Cui et al., 2019; Li et al., 2021) have brought further performance gains.”
• $O(n)$ Connections are Expressive Enough: Universal Approximability of Sparse Transformers (2020, methodology): “To mitigate this issue, researchers have developed methods to sparsify the pairwise interactions/connections in self-attention layers to reduce the computational complexity and/or improve model interpretability, and have shown successful empirical results on tasks with long sequence lengths [2, 6, 8”