Smatch: an Evaluation Metric for Semantic Feature Structures

By Shu Cai, Kevin Knight. 2013

421 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Total Recall: a Customized Continual Learning Method for Neural Semantic Parsers (2021, methodology): “To characterize differences in structures, we first compute similarities between LFs by sim(yi,yj) = (Smatch(yi,yj) + Smatch(yj ,yi))/2, where Smatch (Cai and Knight, 2013) is a asymmetrical similarity score between two LFs yielded by calculating the overlapping percentage of their triples.”
Generating Semantically Precise Scene Graphs from Textual Descriptions for Improved Image Retrieval (2015, methodology): “, 2013) graphs, we use Smatch F1 (Cai and Knight, 2013) as an additional intrinsic metric.”
GPT-too: A Language-Model-First Approach for AMR-to-Text Generation (2020, methodology): “We then re-score each sentence using the standard AMR parsing metric Smatch (Cai and Knight, 2013) by comparing the gold and parsed AMRs.”
L2F/INESC-ID at SemEval-2017 Tasks 1 and 2: Lexical and semantic features in word and textual similarity (2017, background): “The only semantic feature is the SMATCH score (Cai and Knight, 2013) which represents the similarity among two AMR graphs (Banarescu et al., 2013).”
Neural AMR: Sequence-to-Sequence Models for Parsing and Generation (2017, methodology): “We achieve competitive results for the parsing task (SMATCH 62.1) and state-of-the-art performance for generation (BLEU 33.8).”
AMR-to-text Generation with Synchronous Node Replacement Grammar (2017, methodology): “As a next step, we match between pairs of initial rules ri and rj according to Cai and Knight (2013), and generate rij by collapsing ri with rj , if ri contains rj 2 (Line 6-8).”
A Discriminative Graph-Based Parser for the Abstract Meaning Representation (2014, methodology): “, 2013) under the Smatch score (Cai and Knight, 2013), presenting the first published results on automatic AMR parsing.”
Discourse Representation Structure Parsing (2018, methodology): “D-match is based on Smatch 5 , a metric used to evaluate AMR graphs (Cai and Knight, 2013); it calculates F 1 on discourse representation graphs (DRGs), i.e., triples of nodes, arcs, and their referents, applying multiple restarts to obtain a good referent (node) mapping between graphs.”
Broad-coverage CCG Semantic Parsing with AMR (2015, methodology): “Finally, Figure 5 plots development S MATCH F1 with and without early updates.”
SPICE: Semantic Propositional Image Caption Evaluation (2016, background): “Unlike Smatch [27], a recently proposed metric for evaluating AMR parsers that considers multiple alignments of AMR graphs, we make no allowance for partial credit when only one element of a tuple is incorrect.”
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations (2022, unknown): “We investigate whether S MATCH (Cai and Knight, 2013), a metric that measures the degree of overlap between two AMRs, correlates with factuality judgments.”
Toward Abstractive Summarization Using Semantic Representations (2018, methodology): “pdf (3)AMR parse quality is evaluated using smatch (Cai and Knight, 2013), which measures the accuracy of concept and relation predictions.”
Crowdsourcing Question-Answer Meaning Representations (2017, background): “(2016)8: (1) Concept identification, which measures agreement on the nodes of the structures, and (2) an unlabeled variant of the SMATCH score (Cai and Knight, 2013), which measures the agreement on the set of relations between concepts, ignoring their label.”
Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR (2020, background): “Based on IAA estimates by Banarescu et al. (2013), this parser (80.3 Smatch F1 2 ) is almost on-par with human agreement (estimates range between 0.71 and 0.83 Smatch F1).”
Resources for building applications with Dependency Minimal Recursion Semantics (2016, methodology): “The other two matching approaches find the best possible match, even a non-exact one, and assess its quality with an F-score, computed based on the number of matched nodes and links, similarly to a metric by Cai and Knight (2013). Nodes and links are treated as equally important.”