On the Evaluation of Semantic Phenomena in Neural Machine Translation Using Natural Language Inference

By Adam Poliak, Yonatan Belinkov, James R. Glass, Benjamin Van Durme. 2018

42 citations recorded by the source corpus.

We propose a process for investigating the extent to which sentence representations arising from neural machine translation (NMT) systems encode distinct semantic phenomena. We use these representations as features to train a natural language inference (NLI) classifier based on datasets recast from existing semantic annotations. In applying this process to a representative NMT system, we find its encoder appears most suited to supporting inferences at the syntax-semantics interface, as compared to anaphora resolution requiring world knowledge. We conclude with a discussion on the merits and potential deficiencies of the existing process, and how it may be improved and extended as a broader framework for evaluating semantic coverage

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

The Natural Language Decathlon: Multitask Learning as Question Answering (2018, background): “By unifying tasks under a single perspective, it is possible to explore these relationships [Wang et al., 2018, Poliak et al., 2018a,b].”
What do you learn from context? Probing for sentence structure in contextualized word representations (2019, methodology): “In the DNC, these tasks are all recast into natural language inference (NLI) format (White et al., 2017), i.e. systems must understand the targeted semantic phenomenon in order to make correct inferences about en-9 tailment.”
Probing What Different NLP Tasks Teach Machines about Function Word Comprehension (2019, methodology): “Using recast NLI, Poliak et al. (2018a) probe for semantic phenomena in neural machine translation encoders.”
Dialogue Natural Language Inference (2018, methodology): “Thus, the NLI task has been used for learning general sentence representations [4] and for evaluating NLP models [19, 23], with the expectation that such models will be useful in downstream tasks.”
Inherent Disagreements in Human Textual Inferences (2019, methodology): “For the binary tasks (DNC), we use the same three-way thresholds, but consider scores below 16 .”
Interpretability and Analysis in Neural NLP (2020, background): “…controlled behavior studies on language modelling (Gulordava et al., 2018; Linzen et al., 2016a; Mar-vin and Linzen, 2018) or inference tasks (Poliak et al., 2018a,b; White et al., 2017; Kim et al., 2019; McCoy et al., 2019; Ross and Pavlick, 2019), psycholinguistic methods (Ettinger et al.,…”
Lexicosyntactic Inference in Neural Models (2018, methodology): “The experimental setup – speciﬁcally, the idea of UNKing the embedding verb – was inspired by recent work that uses hypothesis-only baselines for a similar purpose (Gururangan et al., 2018; Poliak et al., 2018c; Tsuchiya, 2018).”
RNNs Implicitly Implement Tensor Product Representations (2018, unknown): “One approach is to infer the information encoded in the representations from the system’s behavior on examples targeting speciﬁc representational components, such as semantics (Pavlick, 2017; Dasgupta et al., 2018; Poliak et al., 2018) or syntax (Linzen et al., 2016).”
Collecting Diverse Natural Language Inference Problems for Sentence Representation Evaluation (2018, methodology): “As described in Poliak et al. (2018b), our hypothesis-only model feeds the hypotheses’ encoded representation directly into the MLP.”
Towards a Unified Natural Language Inference Framework to Evaluate Sentence Representations (2018, methodology): “We use the same training settings and hyperparameters as described in Conneau et al. (2017) and Poliak et al. (2018b).”
Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR (2020, methodology): “First practical attempts of assessing sentence quality with semantic assessment have been examined in MT using semantic role labeling (Lo, 2017) or WSD and NLI (Carpuat, 2013; Poliak et al., 2018), in-between lies SPICE that evaluates caption generation via inferred semantic propositions (Anderson e”