Semantics-aware BERT for Language Understanding

By Zhuosheng Zhang, Yuwei Wu, Zhao Hai, Zuchao Li, Shuailiang Zhang, Xi Zhou, Xiang Zhou. 2019

401 citations recorded by the source corpus.

The latest work on language representations carefully integrates contextualized features into language model training, which enables a series of success especially in various machine reading comprehension and natural language inference tasks. However, the existing language representation models including ELMo, GPT and BERT only exploit plain context-sensitive features such as character or word embeddings. They rarely consider incorporating structured semantic information which can provide rich semantics for language representation. To promote natural language understanding, we propose to incorporate explicit contextual semantics from pre-trained semantic role labeling, and introduce an improved language representation model, Semantics-aware BERT (SemBERT), which is capable of explicitly absorbing contextual semantics over a BERT backbone. SemBERT keeps the convenient usability of its BERT precursor in a light fine-tuning way without substantial task-specific modifications. Compared with BERT, semantics-aware BERT is as simple in concept but more powerful. It obtains new state-of-the-art or substantially improves results on ten reading comprehension and language inference tasks.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

Citations in the corpus

Dual Co-Matching Network for Multi-choice Reading Comprehension (2019, methodology): “2019), as using transformer as the contextualized encoder has shown to be very powerful in language representation (Zhang et al. 2019a; Zhou and Zhao ; Luo, Xiao, and Zhao 2019; Xiao et al. 2019).”
Evaluating Semantic Accuracy of Data-to-Text Generation with Natural Language Inference (2020, methodology): “ a premise— and produces one of the possible outputs: the hypothesis is entailed by (follows from) the premise, contradicts the premise, or their relation is neutral. Recently, neural models for NLI (Zhang et al., 2020b;Liu et al.,2019a,b) reached near-human levels of performance and NLI was used fo”
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data (2020, methodology): “Global–GNN (Bogin et al., 2019a) 52.7 EditSQL + B ERT (Zhang et al., 2019a) 57.6 RatSQL (Wang et al., 2019a) 60.9 IRNet + B ERT (Guo et al., 2019) 60.3 + Memory + Coarse-to-Fine 61.9 IRNet V2 + B ERT 63.9 RyanSQL + B ERT (Choi et al., 2020) 66.6 Our System based on TranX (Yin and Neubig, 2018…”
Large pre-trained language models contain human-like biases of what is right and wrong to do (2021, background): “Semantic-BERT [63] or an extension by logic programming modelling moral reasoning [50, 51] could be applied in the future.”
Deep Learning--based Text Classification (2020, background): “NLI, also known as recognizing textual entailment (RTE), predicts whether the meaning of one text can be inferred from another.”
Retrospective Reader for Machine Reading Comprehension (2020, methodology): “…side 1 of MRC models [ Hu et al. , 2019; Back et al. , 2020 ] , though it has been shown that better decoder or better manner of using encoder still has a signiﬁcant impact on MRC performance, no matter how strong the encoder (i.e., the adopted pre-trained LM) it is [ Zhang et al. , 2020a ] .”
When Combating Hype, Proceed with Caution (2021, background): “Looking only to highly-cited works from the last two years that cite Jia and Liang, similar statements can be found in Xu et al. (2020), Zhang et al. (2020), and others.”
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey (2021, methodology): “[218] incorporate SRL predictions into the pre-training procedure of BERT, improving the performance on TE and QA tasks.”
A Dutch Dataset for Cross-lingual Multilabel Toxicity Detection (2021, methodology): “We used sub-word token representation that aligns with the base vocabulary representation (Zhang et al., 2020).”
HUBERT Untangles BERT to Improve Transfer across NLP Tasks (2019, background): “…et al., 2018) has demonstrated great power for providing general-purpose vector embeddings of natural language: its representations have served as the basis of many successful deep Natural Language Processing (NLP) models on a variety of tasks (e.g., Liu et al., 2019a;b; Zhang et al., 2019).”
SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models (2023, background): “This gives the model a strong understanding of language (Jawahar et al., 2019; Raffel et al., 2020), powerful contextual reasoning (Zhang et al., 2020), as well as world knowledge (Liusie et al., 2023).”