SenseBERT: Driving Some Sense into BERT

By Yoav Levine, Barak Lenz, Or Dagan, Ori Ram, Dan Padnos, Or Sharir, Shai Shalev-Shwartz, Amnon Shashua, Yoav Shoham. 2019

203 citations recorded by the source corpus.

The ability to learn from large unlabeled corpora has allowed neural language models to advance the frontier in natural language understanding. However, existing self-supervision techniques operate at the word form level, which serves as a surrogate for the underlying semantic content. This paper proposes a method to employ weak-supervision directly at the word sense level. Our model, named SenseBERT, is pre-trained to predict not only the masked words but also their WordNet supersenses. Accordingly, we attain a lexical-semantic level language model, without the use of human annotation. SenseBERT achieves significantly improved lexical understanding, as we demonstrate by experimenting on SemEval Word Sense Disambiguation, and by attaining a state of the art result on the ‘Word in Context’ task.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Entities as Experts: Sparse Memory Access with Entity Supervision (2020, background): “…with other approaches seeking to inject knowledge into language models (Ahn et al., 2016; Yang et al., 2016; Logan et al., 2019; Zhang et al., 2019; Levine et al., 2019; Xiong et al., 2019a; Peters et al., 2019; Poerner et al., 2019; Wang et al., 2020 Memory Augmented Neural Networks Our entity…”
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters (2020, background): “Recently, some efforts have been made to exploit injecting knowledge into pre-trained language models (Zhang et al., 2019; Lauscher et al., 2019; Levine et al., 2019; Peters et al., 2019; He et al., 2019; Xiong et al., 2020).”
Pre-trained models for natural language processing: A survey (2020, background): “The external knowledge ranges from linguistic [94, 83, 136, 191], semantic [99], commonsense [52], factual [214, 136, 111, 202, 195], to domain-specific knowledge [58, 111].”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “Some of the examples of knowledge enriched T-PTLMs are CasualBERT [197], KnowBERT [189], SenseBERT [204], LIMIT-BERT [199], LiBERT [198], SentiLARE [200], ERNINE [188], E-BERT [186] in the general domain and Clincal Kb-BERT [201], Clinical Kb-ALBERT [201], SapBERT [202], Sap-XLMR [202], UmlsBERT [12”
Better Language Model with Hypernym Class Prediction (2022, methodology): “More recently, Levine et al. (2020) pretrain masked LMs (Devlin et al., 2019) by predicting WordNet supersense labels.”
Towards Generalizable Neuro-Symbolic Systems for Commonsense Question Answering (2019, background): “Peters et al. (2019) and Levine et al. (2019) both tried to injecting knowlegde into language models by pretraining on knowledge bases.”