The Meaning Factory: Formal Semantics for Recognizing Textual Entailment and Determining Semantic Similarity

By Johannes Bjerva, Johan Bos, Rob van der Goot, Malvina Nissim. 2014

93 citations recorded by the source corpus.

Shared Task 1 of SemEval-2014 comprised two subtasks on the same dataset of sentence pairs: recognizing textual entailment and determining textual similarity. We used an existing system based on formal semantics and logical inference to participate in the first subtask, reaching an accuracy of 82%, ranking in the top 5 of more than twenty participating systems. For determining semantic similarity we took a supervised approach using a variety of features, the majority of which was produced by our system for recognizing textual entailment. In this subtask our system achieved a mean squared error of 0.322, the best of all participating systems.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Skip-Thought Vectors (2015, neutral): “[19] 0.8070 0.7489 0.3550 Meaning Factory [20] 0.8268 0.7721 0.3224 ECNU [21] 0.8414 – –”
SemEval-2014 Task 1: Evaluation of Compositional Distributional Semantic Models on Full Sentences through Semantic Relatedness and Textual Entailment (2014, background): “In the table, systems in bold are those for which the authors submitted a paper (Ferrone and Zanzotto, 2014; Bjerva et al., 2014; Beltagy et al., 2014; Lai and Hockenmaier, 2014; Alves et al., 2014; León et al., 2014; Bestgen, 2014; Zhao et al., 2014; Vo et al., 2014; Biçici and Way, 2014; Lien…”
Siamese Recurrent Architectures for Learning Sentence Similarity (2016, neutral): “Another high-performing system, from Bjerva et al. (2014), uses formal semantics and logical inference in conjunction with features derived from the popular word2vec neural language model of Mikolov et al. (2013), whose word vectors we employ as the sole input to our model.”
Pairwise Word Interaction Modeling with Deep Neural Networks for Semantic Similarity Measurement (2016, neutral): “3550 Bjerva et al. (2014) 0.”
From Paraphrase Database to Compositional Paraphrase Model and Back (2015, methodology): “The PPDB has recently been used for monolingual alignment (Yao et al., 2013), for predicting sentence similarity (Bjerva et al., 2014), and to improve the coverage of FrameNet (Rastogi and Van Durme, 2014).”
PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification (2015, methodology): “It has been used for recognizing textual entailment (Beltagy et al., 2014; Bjerva et al., 2014), measuring the semantic similarity of texts (Han et al.”
DiSAN: Directional Self-Attention Network for RNN/CNN-free Language Understanding (2017, neutral): “(Bjerva et al. 2014), (Zhao, Zhu, and Lan 2014), (Socher et al.”
Bi-Directional Block Self-Attention for Fast and Memory-Efficient Sequence Modeling (2018, neutral): “Model Pearson’s r Spearman’s ρ MSE Meaning Factory (Bjerva et al., 2014) 0.”
SemEval-2015 Task 1: Paraphrase and Semantic Similarity in Twitter (PIT) (2015, background): “Rob (van der Goot and van Noord, 2015): This system is inspired by a state-of-the-art semantic relatedness prediction system by Bjerva et al. (2014).”
MonaLog: a Lightweight System for Natural Language Inference Based on Monotonicity (2019, result): “Second, we want to make our results comparable to those of previous logic-based models such as the ones described in ( Bjerva et al., 2014; Abzianidze; Mart´ınez-G´omez et al., 2017; 5 https://github.com/google-research/bert Yanaka et al., 2018), which were also tested on SICK.”
A Continuously Growing Dataset of Sentential Paraphrases (2017, background): “, 2015), textual entailment (Dagan et al., 2006; Bjerva et al., 2014; Marelli et al., 2014; Izadinia et al., 2015), vector semantics (Faruqui et al.”
A Natural Proof System for Natural Language (2017, methodology): “When compared to the Meaning Factory, the system with high precision, LangPro allows three times less false positives—classifying neutral problems as entailment or contradiction.”
SICK through the SemEval glasses. Lesson learned from the evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment (2016, background): “Interestingly, some participants used the non-primary runs to explore the performance level that could be reached by exploiting weaknesses in the data that are not likely to hold in future tasks of the same kind (for instance, the non-primary run 3 submitted by The Meaning Factory exploited sentence”
Learning to Compose Words into Sentences with Reinforcement Learning (2016, methodology): “Some of these models (Lai & Hockenmaier, 2014; Jimenez et al., 2014; Bjerva et al., 2014) rely on feature engineering and are designed speciﬁcally for this task.”
Most “babies” are “little” and most “problems” are “huge”: Compositional Entailment in Adjective-Nouns (2016, unknown): “There have been an enormous range of approaches to automatic RTE– from those based on theorem proving (Bjerva et al., 2014) to those based on vector space models of semantics (Bowman et al., 2015a).”