NUBIA: NeUral Based Interchangeability Assessor for Text Generation

By Hassan Kané, Muhammed Yusuf Kocyigit, Ali Abdalla, Pelkins Ajanoh, Mohamed Coulibali. 2020

68 citations recorded by the source corpus.

We present NUBIA, a methodology to build automatic evaluation metrics for text generation using only machine learning models as core components. A typical NUBIA model is composed of three modules: a neural feature extractor, an aggregator and a calibrator. We demonstrate an implementation of NUBIA showing competitive performance with stateof-the art metrics used to evaluate machine translation and state-of-the art results for image captions quality evaluation. In addition to strong performance, NUBIA models have the advantage of being modular and improve in synergy with advances in text generation models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Natural Language Generation

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “…distance (Kusner et al., 2015), which relies on non-contextual word embeddings, and BERT-SCORE (Zhang et al., 2020), which aggregates cosine distances between represented tokens in a sequence, among others (Zhao et al., 2019; Clark et al., 2019; Kane et al., 2020; Colombo et al., 2021, inter alia).”
Chain of Explanation: New Prompting Method to Generate Quality Natural Language Explanation for Implicit Hate Speech (2022, background): “The NUBIA metric, known as the most advanced metric, reflects how interchangeable the sentences are using various neural models [15].”
A Survey of Evaluation Metrics Used for NLG Systems (2020, methodology): “The BLEURT rating is obtained using a linear layer on the embedding produced for the prepended [CLS] token: NUBIA: Kané et al. [58] propose a three-stage architecture called NUBIA for NLG evaluation.”
FrugalScore: Learning Cheaper, Lighter and Faster Evaluation Metrics for Automatic Text Generation (2021, methodology): “NUBIA (Kane et al., 2020) consists of three modules: a feature extractor based on RoBERTa (Liu et al., 2019) and GPT-2 (Radford et al., 2019) fine-tuned on language evaluation tasks, an aggregator trained to predict the quality of the hypothesis given the reference using the extracted features, and…”
CLIPScore: A Reference-free Evaluation Metric for Image Captioning (2021, background): “(Cui et al., 2018) 46.6 BERT-S++ (Yi et al., 2020) 46.7 TIGEr (Jiang et al., 2019) 49.3 NUBIA * (Kane et al., 2020) 49.5 ViLBERTScore-F (Lee et al., 2020)”