Portuguese Named Entity Recognition using BERT-CRF
By Fábio Souza, Rodrigo Nogueira, Roberto de Alencar Lotufo. 2019
299 citations recorded by the source corpus.
Recent advances in language representation using neural networks have made it viable to transfer the learned internal states of a trained model to downstream natural language processing tasks, such as named entity recognition (NER) and question answering. It has been shown that the leverage of pre-trained language models improves the overall performance on many tasks and is highly beneficial when labeled data is scarce. In this work, we employ a pre-trained BERT with Conditional Random Fields (CRF) architecture to the NER task on the Portuguese language, combining the transfer capabilities of BERT with the structured predictions of CRF. We explore feature-based and fine-tuning training strategies for the BERT model. Our fine-tuning approach obtains new state-of-the-art results on the HAREM I dataset, improving the F1-score by 3.2 points on the selective scenario (5 NE classes) and by 3.8 points on the total scenario (10 NE classes).
Subfields: Artificial Intelligence, Management Science and Operations Research
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Frank K. Soong
- Lei He
- Carolina Scarton
- Alexander Allauzen
- Benjamin Lecouteux
- Benoit Crabb'e
- Hojin Kang
- Vincent Segonne
In Management Science and Operations Research
- Benjamin Lecouteux
- Rodrigo Nogueira
- Laurent Besacier
- Ricardo Henao
- Frank K. Soong
- Kalina Bontcheva
- Diego Furtado Silva
- Alexander Allauzen
Citations in the corpus
- Few-Shot Class-Incremental Learning for Named Entity Recognition (2022, methodology): “Following Beltagy et al. (2019); Souza et al. (2019), we use the BERT-CRF as our NER model, which consists of a BERT base (Devlin et al., 2018) encoder with a linear projection and a conditional random field (CRF) (Lafferty et al., 2001) layer for prediction.”
- FlauBERT: Unsupervised Language Model Pre-training for French (2019, background): “…Arabic (Antoun et al., 2020), Dutch (de Vries et al., 2019; Delobelle et al., 2020), Finnish (Virtanen et al., 2019), Italian (Polignano et al., 2019), Portuguese (Souza et al., 2019), Russian (Kuratov and Arkhipov, 2019), Spanish (Cañete et al., 2020), and Vietnamese (Nguyen and Nguyen, 2020).”
- PTT5: Pretraining and validating the T5 model on Brazilian Portuguese data (2020, background): “For Portuguese tasks, BERT models have already shown improved performance when pretrained on a Portuguese corpus [20].”
- Improving Prosody with Linguistic and Bert Derived Features in Multi-Speaker Based Mandarin Chinese Neural TTS (2020, methodology): “Similar Bert-CRF model has been applied to tasks like Named Entity Recognition (NER) [23] and sentiment analysis [24].”
- Spanish Pre-trained BERT Model and Evaluation Data (2023, methodology): “This has led to the release of BERT models in Russian (Kuratov & Arkhipov, 2019), French (Martin et al., 2019; Le et al., 2019), Dutch (de Vries et al., 2019; Delobelle et al., 2020), Italian (Polignano et al., 2019), and Por-tugese (Souza et al., 2019).”
- Toxic Language Detection in Social Media for Brazilian Portuguese: New Dataset and Multilingual Analysis (2020, methodology): “Evaluation is done in terms of Hamming loss and average precision (Tsoumakas et al., 2009).”