Does It Capture STEL? A Modular, Similarity-based Linguistic Style Evaluation Framework

By Anna Wegmann, Dong Nguyen. 2021

20 citations recorded by the source corpus.

Style is an integral part of natural language. However, evaluation methods for style measures are rare, often task-specific and usually do not control for content. We propose the modular, fine-grained and content-controlled similarity-based STyle EvaLuation framework (STEL) to test the performance of any model that can compare two sentences on style. We illustrate STEL with two general dimensions of style (formal/informal and simple/complex) as well as two specific characteristics of style (contrac’tion and numb3r substitution). We find that BERT-based methods outperform simple versions of commonly used style measures like 3-grams, punctuation frequency and LIWC-based approaches. We invite the addition of further tasks and task instances to STEL and hope to facilitate the improvement of style-sensitive measures.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “…are trained on more human ratings, and introduce initialization and and pretraining schemes (Sellam et al., 2020; Rei et al., 2020; Pu et al., 2021; Wegmann and Nguyen, 2021, inter alia), or focus on specific aspects like the faithfulness of generated text (e.g., Kryscinski et al., 2020;…”