Pooled Contextualized Embeddings for Named Entity Recognition

By Alan Akbik, Tanja Bergmann, Roland Vollgraf. 2019

273 citations recorded by the source corpus.

Contextual string embeddings are a recent type of contextualized word embedding that were shown to yield state-of-the-art results when utilized in a range of sequence labeling tasks. They are based on character-level language models which treat text as distributions over characters and are capable of generating embeddings for any string of characters within any textual context. However, such purely character-based approaches struggle to produce meaningful embeddings if a rare string is used in a underspecified context. To address this drawback, we propose a method in which we dynamically aggregate contextualized embeddings of each unique string that we encounter. We then use a pooling operation to distill a ”global” word representation from all contextualized instances. We evaluate these ”pooled contextualized embeddings” on common named entity recognition (NER) tasks such as CoNLL-03 and WNUT and show that our approach significantly improves the state-of-the-art for NER. We make all code and pre-trained models available to the research community for use and reproduction.

Subfields: Artificial Intelligence, Natural Language Processing, Information Extraction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Pengfei Liu
2. Graham Neubig
3. Maosong Sun
4. Zhiyuan Liu
5. Yixin Liu
6. Jinlan Fu
7. Mohit Bansal
8. Anna Rogers

In Natural Language Processing

1. Pengfei Liu
2. Graham Neubig
3. Jinlan Fu
4. Xuanjing Huang
5. Maosong Sun
6. Zhiyuan Liu
7. Yixin Liu
8. Fei Huang

In Information Extraction

1. Pengfei Liu
2. Jinlan Fu
3. Graham Neubig
4. Xuanjing Huang
5. Fei Huang
6. Nguyen Bach
7. Yong Jiang
8. Kewei Tu

Citations in the corpus

• HuggingFace's Transformers: State-of-the-art Natural Language Processing (2019, neutral): “The first direction includes Natural Language Processing libraries such as AllenNLP 13 (Gardner et al., 2018), SpaCy 14 (Honnibal and Montani, 2017), flair 15 (Akbik et al., 2018, 2019) or PyText 16 .”
• A Primer in BERTology: What We Know About How BERT Works (2020, methodology): “The methods to distill a contextualized representation into static include aggregating the information across multiple contexts (Akbik et al., 2019; Bommasani et al., 2020), encoding "semantically bleached" sentences that rely almost exclusively on the meaning of a given word (e.”
• MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification (2020, background): “The pre-training and fine-tuning framework has achieved huge success on NLP applications in recent years, and has been applied to a variety of NLP tasks (Radford et al., 2018; Chen et al., 2019; Akbik et al., 2019).”
• FLAIR: An Easy-to-Use Framework for State-of-the-Art NLP (2019, background): “, 2018) yes PooledFlairEmbeddings pooled version of FLAIR embeddings (Akbik et al., 2019b) yes”
• ExplainaBoard: An Explainable Leaderboard for NLP (2021, background): “, 2020), FLERT (Schweter and Akbik, 2020) and FLAIR (Akbik et al., 2019).”
• How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence (2020, background): “…from texts, and there are many NLP works which concentrate on IE, including name entity recognition (Lample et al., 2016; Kuru et al., 2016; Akbik et al., 2019), relation extraction (Zeng et al., 2015; Miwa and Bansal, 2016; Lin et al., 2016; Christopoulou et al., 2018), and event…”
• Improved Differentiable Architecture Search for Language Modeling and Named Entity Recognition (2019, methodology): “A s-0 Model F1 best published BiLSTM-CRF (Lample et al., 2016) 90.94 BiLSTM-CRF+ELMo (Peters et al., 2018) 92.22 BERT Base (Devlin et al., 2018) 92.40 BERT Large (Devlin et al., 2018) 92.80 BiLSTM-CRF+PCE (Akbik et al., 2019) 93 trand of NAS research focuses on reinforcement learning (Zoph and Le, 2”
• Named Entity Recognition as Dependency Parsing (2020, unknown): “In addition, we also tested our system on the revised version of German data to compare with the model by Akbik et al. (2018), our system again achieved a substantial gain of 2% when compared with their system.”
• Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning (2021, methodology): “Most of the work takes NER as a sequence labeling problem and applies the linear-chain CRF (Lafferty et al., 2001) to achieve state-of-the-art accuracy (Ma and Hovy, 2016; Lample et al., 2016; Akbik et al., 2018, 2019; Wang et al., 2020b).”
• Structure-Level Knowledge Distillation For Multilingual Sequence Labeling (2020, methodology): “On the tasks of NER and POS tagging, Flair embeddings (Akbik et al., 2018, 2019) is a state-of-the-art method based on character-level language models.”
• SpanNER: Named Entity Re-/Recognition as Span Prediction (2021, methodology): “NER as Different Tasks Although NER is commonly formulated as a sequence labeling task (Chiu and Nichols, 2015; Huang et al., 2015; Ma and Hovy, 2016; Lample et al., 2016; Akbik et al., 2018; Peters et al., 2018; Devlin et al., 2018; Xia et al., 2019; Akbik et al., 2019; Luo et al., 2020; Lin et al.”
• Interpretable Multi-dataset Evaluation for Named Entity Recognition (2020, methodology): “…2016; Collobert et al., 2011) and learning of pre-trained embeddings (Peters et al., 2018; Akbik et al., 2018, 2019; Devlin et al., 2018; Pennington et al., 2014), Named Entity Recognition (NER) systems are evolving rapidly but also quickly reaching a performance plateau (Akbik et al., 2018, 2019).”
• Named Entity Recognition for Social Media Texts with Semantic Augmentation (2020, unknown): “09 Akbik et al. (2019) - 49.”