A Comparison of Greedy and Optimal Assessment of Natural Language Student Input Using Word-to-Word Similarity Metrics

By Vasile Rus, Mihai C. Lintean. 2012

158 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization (2018, background): “The Greedy metric yields the maximum cosine similarity over embeddings of two utterances [41].”
BERTScore: Evaluating Text Generation with BERT (2019, background): “Rus and Lintean (2012) also study greedy word embedding matching.”
A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues (2016, background): “age), Embedding Extrema (Extrema) and Embedding Greedy (Greedy) (Mitchell and Lapata 2008; Forgues et al. 2014; Rus and Lintean 2012).”
USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation (2020, background): “Greedy Matching (Rus and Lintean, 2012) is an embedding-based metric that greedily matches each word in the generated sequence to a reference word based on the cosine similarity of their embeddings.”
Generating Factoid Questions With Recurrent Neural Networks: The 30M Factoid Question-Answer Corpus (2016, methodology): “We therefore also make use of a sentence similarity metric, as proposed by Rus and Lintean (2012), which we will denote Embedding Greedy (Emb.”
A Survey of Evaluation Metrics Used for NLG Systems (2020, methodology): “ the hypothesis and the reference(s). We discuss such word embedding based metrics in this subsection. In all the discussion that follows, we represent the embedding of a word�as →−�. Greedy Matching [134]: This metric considers each token in the reference and greedily matches it to the closest toke”
Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog Evaluation (2020, background): “, 2002; Lin, 2004; Banerjee and Lavie, 2005), embedding based (Rus and Lintean, 2012; Forgues et al., 2014), or learning based (Tao et al.”
Evaluating Style Transfer for Text (2019, background): “It repeats the process in the reverse direction and takes the average of those two scores (Rus and Lintean, 2012).”
A Hierarchical Latent Structure for Variational Conversation Modeling (2018, background): “The embedding-based metrics (Serban et al., 2017; Rus and Lintean, 2012) measure the textual similarity between the words in the model response and the ground truth.”
Emergent: a novel data-set for stance classification (2016, methodology): “Therefore, based on the work by Rus and Lintean (2012) we compute an alignment using the Paraphrase Database (PPDB) (Pavlick et al., 2015) and the Kuhn-Munkres algo-rithm (Kuhn, 1955; Munkres, 1957) as follows.”
Neural Response Generation via GAN with an Approximate Embedding Layer (2017, background): “The Greedy metric is to greedily match words in two given utterances based on the cosine similarities of their embeddings, and to average the obtained (Rus and Lintean, 2012).”
Improving Dialog Evaluation with a Multi-reference Adversarial Dataset and Large Scale Pretraining (2020, background): “, 2002; Banerjee and Lavie, 2005; Lin, 2004) or embedding based metrics (Forgues et al., 2014; Rus and Lintean, 2012; Zhang et al., 2020a) to compare the model’s response with a single reference.”
Active Evaluation: Efficient NLG Evaluation with Few Pairwise Comparisons (2022, background): “Out of the total 35 systems, we held out of1062 7 systems and only used the remaining 28 systems.1063 Summarization: We select 11 systems that have1064 human annotations between each pair of them.1065 These systems are GPT3-like models with varying1066 model sizes (3B, 6B, 12B) and training strategi”
DialogWAE: Multimodal Response Generation with Conditional Wasserstein Auto-Encoder (2018, background): “Greedy : greedily matching words in two utterances based on the cosine similarities between their embed-dings, and to average the obtained scores (Rus and Lintean, 2012).”
Dialogue Generation: From Imitation Learning to Inverse Reinforcement Learning (2018, background): “• Greedy embedding: This metric relies on cosine-similarity but adopts greedy matching to find the closest word in the target response for each word in the generated response (Rus and Lintean 2012).”
Perturbation CheckLists for Evaluating NLG Evaluation Metrics (2021, methodology): “(Refer appendix D for full plots) et al., 2015), Embedding Averaging (EAvg) (Lan-dauer and Dumais, 1997), Greedy Matching (GM) (Rus and Lintean, 2012), and Vector Extrema (VE) (Forgues and Pineau, 2014), we use the implementation provided by Sharma et al. (2017).”