MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare

By Shaoxiong Ji, Tianlin Zhang, Luna Ansari, Jie Fu, Prayag Tiwari, Erik Cambria. International Conference on Language Resources and Evaluation, 2021

418 citations recorded by the source corpus.

Mental health is a critical issue in modern society, and mental disorders could sometimes turn to suicidal ideation without adequate treatment. Early detection of mental disorders and suicidal ideation from social content provides a potential way for effective social intervention. Recent advances in pretrained contextualized language representations have promoted the development of several domainspecific pretrained models and facilitated several downstream applications. However, there are no existing pretrained language models for mental healthcare. This paper trains and release two pretrained masked language models, i.e., MentalBERT and MentalRoBERTa, to benefit machine learning for the mental healthcare research community. Besides, we evaluate our trained domain-specific models and several variants of pretrained language models on several mental disorder detection benchmarks and demonstrate that language representations pretrained in the target domain improve the performance of mental health detection tasks.

Subfields: Applied Psychology, Artificial Intelligence, Social Psychology

View Paper: https://www.semanticscholar.org/paper/9442019231e11d3e6eb3d4729dcaad7d7871c62c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Social Psychology

Citations in the corpus

Gendered Mental Health Stigma in Masked Language Models (2022, methodology): “Our project expands upon this work as we focus on more recent MLMs, including general-purpose MLM RoBERTa, as well as MLMs pretrained on health and mental health corpora, MentalRoBERTa (Ji et al., 2021) and ClinicalLongformer (Li et al.”
A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, neutral): “BioBERT [91] FT PubMed, PMC Biomedical NER, RE, QA BlueBert [130] FT PubMed, MIMIC-III BLUE MIMIC-BERT [131] FT MIMIC-III Biomedical NER BioFLAIR [132] FT PubMed Bio NER Bio-ELECTRA-small [133] PT PubMed Biomedical NER AlphaBERT [134] FT Discharge diagnoses Extractive Summarization Task Spanish-bert”
Understanding the Effects of Iterative Prompting on Truthfulness (2024, background): “This prowess has rendered them invaluable across diverse sectors, including finance, healthcare, and autonomous systems, revolutionizing conventional approaches to tasks in these domains [4–7].”
Leveraging Large Language Models for Mental Health Prediction via Online Text Data (2023, methodology): “Meanwhile, Mental-Alpaca can further perform on par with the task-specific state-of-the-art Mental-RoBERTa [42].”
MAQuA: Adaptive Question-Asking for Multidimensional Mental Health Screening using Item Response Theory (2025, unknown): “…and empty response list responses ← ∅ 3: while responses , and stopping rule not met on θ do 4: Select next item p to maximize information at current θ 5: Present prompt p and capture free-text response t 6: Compute discrete response score s = NLP score ( t ) 7: Append ( p, s ) to responses 8:…”
MAQuA: Multi-outcome Adaptive Question-Asking for Mental Health using Item Response Theory (2026, unknown): “However, comprehensive evaluations indicate that such LLM-based assessments are inconsistent (Ji et al., 2022) and, in many cases, less accurate than dedicated condition-specific models with established psychometric validity (Harri-gian et al., 2020).”