COVID-Twitter-BERT: A natural language processing model to analyse COVID-19 content on Twitter

By Martin Müller, Marcel Salathé, Per Egil Kummervold. Frontiers in Artificial Intelligence, 2020

413 citations recorded by the source corpus.

Introduction This study presents COVID-Twitter-BERT (CT-BERT), a transformer-based model that is pre-trained on a large corpus of COVID-19 related Twitter messages. CT-BERT is specifically designed to be used on COVID-19 content, particularly from social media, and can be utilized for various natural language processing tasks such as classification, question-answering, and chatbots. This paper aims to evaluate the performance of CT-BERT on different classification datasets and compare it with BERT-LARGE, its base model. Methods The study utilizes CT-BERT, which is pre-trained on a large corpus of COVID-19 related Twitter messages. The authors evaluated the performance of CT-BERT on five different classification datasets, including one in the target domain. The model's performance is compared to its base model, BERT-LARGE, to measure the marginal improvement. The authors also provide detailed information on the training process and the technical specifications of the model. Results The results indicate that CT-BERT outperforms BERT-LARGE with a marginal improvement of 10-30% on all five classification datasets. The largest improvements are observed in the target domain. The authors provide detailed performance metrics and discuss the significance of these results. Discussion The study demonstrates the potential of pre-trained transformer models, such as CT-BERT, for COVID-19 related natural language processing tasks. The results indicate that CT-BERT can improve the classification performance on COVID-19 related content, especially on social media. These findings have important implications for various applications, such as monitoring public sentiment and developing chatbots to provide COVID-19 related information. The study also highlights the importance of using domain-specific pre-trained models for specific natural language processing tasks. Overall, this work provides a valuable contribution to the development of COVID-19 related NLP models.

Subfields: Artificial Intelligence, Sociology and Political Science

View Paper: https://www.semanticscholar.org/paper/126fb7df6bcab2b70000dfe5b940ada63ae1ba6a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Erik Cambria
2. Xisen Jin
3. Xiang Ren
4. Qika Lin
5. Henghui Zhu
6. Xiang Lan
7. Diksha Khurana
8. Rui Mao

In Sociology and Political Science

1. Erik Cambria

Citations in the corpus

• Natural language processing: state of the art, current trends and challenges (2017, methodology): “[90] used the BERT model to analyze the tweets on covid-19 content.”
• Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora (2021, background): “Fine-tuning from a static and possibly “outdated" PTLM may limit the model performance on downstream tasks, as the PTLM may no longer provide an effective model initialization (Beltagy et al., 2019; Müller et al., 2020).”
• A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, methodology): “…HES Disease Prediction BioMed-RoBERTa [138] FT BIOMED CHEMPROT, RCT RadBERT [139] FT Radiology Report Corpus Report Coding, Summarization CT-BERT [140] FT Tweet COVID-19 Text Classification French-BERT [141] FT French clinical documents DEFT challenge FS-/RAD-/GER-BERT [142] FT,PT Unstructured…”