Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attention

By Yichong Xu, Chenguang Zhu, Shuohang Wang, Siqi Sun, Hao Cheng, Xiaodong Liu, Jianfeng Gao, Pengcheng He, Michael Zeng, Xuedong Huang. 2021

64 citations recorded by the source corpus.

Most of today's AI systems focus on using self-attention mechanisms and transformer architectures on large amounts of diverse data to achieve impressive performance gains. In this paper, we propose to augment the transformer architecture with an external attention mechanism to bring external knowledge and context to bear. By integrating external information into the prediction process, we hope to reduce the need for ever-larger models and increase the democratization of AI systems. We find that the proposed external attention mechanism can significantly improve the performance of existing AI systems, allowing practitioners to easily customize foundation AI models to many diverse downstream applications. In particular, we focus on the task of Commonsense Reasoning, demonstrating that the proposed external attention mechanism can augment existing transformer models and significantly improve the model's reasoning capabilities. The proposed system, Knowledgeable External Attention for commonsense Reasoning (KEAR), reaches human parity on the open CommonsenseQA research benchmark with an accuracy of 89.4% in comparison to the human accuracy of 88.9%.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, methodology): “The previous SoTA baselines are obtained from: a : DeBERTaV3-large + KEAR (Xu et al., 2021b), b : Chowdhery et al. (2022), c : UniﬁedQA-FT (Khashabi et al., 2020).”
The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpus (2021, background): “…NLP tasks, while large pretrained models have achieved remarkable performance (Raffel et al., 2019; Brown et al., 2020; Lourie et al., 2021), researchers have been seeking external repositories of common sense to boost performance further (Mitra et al., 2019; Lin et al., 2021a; Xu et al., 2021).”
STaR: Bootstrapping Reasoning With Reasoning (2022, methodology): “GPT-3 Direct Finetuned [29] 73.0 100 Few-shot Direct GPT-J 20.9 ∼ 0 Few-shot CoT GPT-J 3 36.6 ∼ 0 Few-shot CoT LaMDA 137B [6] 55.”
PaLM 2 Technical Report (2023, neutral): “, 2020), eDeBERTaV3-large+KEAR (Xu et al., 2022), f PaLM+CoT (Suzgun et al.”
Deep Bidirectional Language-Knowledge Graph Pretraining (2022, unknown): “…performance improvements on a broader set of NLP tasks including QA, reasoning and text classification. answering (Lin et al. 2019; Feng et al. 2020; Lv et al. 2020; Wang et al. 2022; Mihaylov & Frank 2018; Yang et al. 2019; Sun et al. 2018, 2019a; Yan et al. 2021; Sun et al. 2022; Xu et al. 2022).”