TENER: Adapting Transformer Encoder for Named Entity Recognition

By Hang Yan, Bo Deng, Xiaonan Li, Xipeng Qiu. arXiv.org, 2019

302 citations recorded by the source corpus.

The Bidirectional long short-term memory networks (BiLSTM) have been widely used as an encoder in models solving the named entity recognition (NER) task. Recently, the Transformer is broadly adopted in various Natural Language Processing (NLP) tasks owing to its parallelism and advantageous performance. Nevertheless, the performance of the Transformer in NER is not as good as it is in other NLP tasks. In this paper, we propose TENER, a NER architecture adopting adapted Transformer Encoder to model the character-level features and word-level features. By incorporating the direction and relative distance aware attention and the un-scaled attention, we prove the Transformer-like encoder is just as effective for NER as other NLP tasks.

Subfields: Artificial Intelligence, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/d6b414487787d0b6efd735a3236a690ad13aae70

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xipeng Qiu
2. Jiashi Feng
3. Shuicheng Yan
4. Xinchao Wang
5. Weihao Yu
6. Fangzhao Wu
7. Chuhan Wu
8. Xuanjing Huang

In Management Science and Operations Research

1. Xipeng Qiu
2. Hang Yan
3. Shuicheng Yan
4. Bo Dai
5. Chuhan Wu
6. Donghong Ji
7. Yongfeng Huang
8. Xuanjing Huang

Citations in the corpus

• FLAT: Chinese NER Using Flat-Lattice Transformer (2020, methodology): “Our model outperforms TENER (Yan et al., 2019) by 1.72 in average F1 score.”
• Unified Named Entity Recognition as Word-Word Relation Classification (2021, background): “Current mainstream work combine the CRF (Lafferty et al. 2001; Finkel et al. 2005) with neural architecture, such as CNN (Collobert et al. 2011; Strubell et al. 2017), bi-directional LSTM (Huang et al. 2015; Lample et al. 2016), and Trans-former (Yan et al. 2019; Li et al. 2020b).”
• A Survey of Transformers (2021, methodology): “, machine translation [35, 91, 104, 123, 137], language modeling [24, 103, 111, 122] and named entity recognition [80, 154].”
• A Unified Generative Framework for Various NER Subtasks (2021, methodology): “…labelling formulation, which will assign a tag to each token in the sentence, has been widely used in the flat NER field (McCal-lum and Li, 2003; Collobert et al., 2011; Huang et al., 2015; Chiu and Nichols, 2016; Lample et al., 2016; Strakov´a et al., 2019; Yan et al., 2019; Li et al., 2020a).”
• MetaFormer Baselines for Vision (2022, background): “Transformer, since being introduced in [73], has become a popular backbone for various tasks in NLP [53, 3, 44, 79, 55], computer vision [5, 17, 69, 84, 74, 45] and other domains [18, 51, 39, 32, 29, 38, 7].”
• Named Entity Recognition for Social Media Texts with Semantic Augmentation (2020, methodology): “To further improve NER performance, we also attempt multiple types of pre-trained word embeddings for feature extraction, which has been demonstrated to be effective in previous studies (Akbik et al., 2018; Jie and Lu, 2019; Kasai et al., 2019; Kim et al., 2019; Yan et al., 2019).”
• Mixed Transformer U-Net for Medical Image Segmentation (2021, background): “It improves the model performance for explicitly providing relative relation, which the ordinary absolute positional embedding cannot [16].”
• DA-Transformer: Distance-aware Transformer (2020, methodology): “(4) TENER (Yan et al., 2019), a variant of Transformer that uses direction-and distance-aware position encoding for named entity recognition.”