Next Sentence Prediction helps Implicit Discourse Relation Classification within and across Domains

By Vera Demberg. 2019

104 citations recorded by the source corpus.

Implicit discourse relation classification is one of the most difficult tasks in discourse parsing. Previous studies have generally focused on extracting better representations of the relational arguments. In order to solve the task, it is however additionally necessary to capture what events are expected to cause or follow each other. Current discourse relation classifiers fall short in this respect. We here show that this shortcoming can be effectively addressed by using the bidirectional encoder representation from transformers (BERT) proposed by Devlin et al. (2019), which were trained on a next-sentence prediction task, and thus encode a representation of likely next sentences. The BERT-based model outperforms the current state of the art in 11-way classification by 8% points on the standard PDTB dataset. Our experiments also demonstrate that the model can be successfully ported to other domains: on the BioDRB dataset, the model outperforms the state of the art system around 15% points.

Subfields: Artificial Intelligence, Natural Language Processing, Information Extraction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Information Extraction

Citations in the corpus

MERLOT: Multimodal Neural Script Knowledge Models (2021, background): “Past work has considered unshufﬂing image-caption pairs [2], but we take a slightly different approach in this work to avoid language-only biases, which can rely on discursive clues to order text [27, 102].”
Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT (2020, background): “In particular, we observe a decent accuracy in identifying discourse relations between adjacent EDUs, perhaps due to the “next sen-Model tence prediction” task in pre-training, as pointed out in (Shi and Demberg, 2019).”
Open-Source Embedding Models: A Comprehensive Survey of Techniques, Benchmarks, and Applications (2026, unknown): “…contextualized word embeddings by analyzing the entire sequence of words in both left and right directions, leveraging the encoder segment of the transformer architecture and being pre-trained on large-scale text corpora using masked language modeling and next sentence prediction objectives [38].”