Combining Independent Modules to Solve Multiple-choice Synonym and Analogy Problems

By Peter D. Turney, Michael L. Littman, Jeffrey P. Bigham, Victor Shnayder. 2003

171 citations recorded by the source corpus.

Existing statistical approaches to natural language problems are very coarse approximations to the true complexity of language processing. As such, no single technique will be best for all problem instances. Many researchers are examining ensemble methods that combine the output of successful, separately developed modules to create more accurate solutions. This paper examines three merging rules for combining probability distributions: the well known mixture rule, the logarithmic rule, and a novel product rule. These rules were applied with state-of-the-art results to two problems commonly used to assess human mastery of lexical semantics|synonym questions and analogy questions. All three merging rules result in ensembles that are more accurate than any of their component modules. The dierences among the three rules are not statistically signicant, but it is suggestive that the popular mixture rule is not the best rule for either of the two problems.

Subfields: Artificial Intelligence, Information Retrieval & Search, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Retrieval & Search

In Natural Language Processing

Citations in the corpus

Language Models are Few-Shot Learners (2020, methodology): “ern-matching and computation. 3.9.3 SAT Analogies To test GPT-3 on another task that is somewhat unusual relative to the typical distribution of text, we collected a set of 374 “SAT analogy” problems [TLBS03]. Analogies are a style of multiple choice question that constituted a section of the SAT co”
From Frequency to Meaning: Vector Space Models of Semantics (2010, methodology): “Several approaches to measuring the semantic similarity of words combine a VSM with a lexicon (Turney et al., 2003; Pantel, 2005; Patwardhan & Pedersen, 2006; Mohammad & Hirst, 2006).”
Domain and Function: A Dual-Space Model of Semantic Relations and Compositions (2012, methodology): “At first the patterns in these pair–pattern matrices were generated by hand (Turney et al., 2003; Turney & Littman, 2005), but later work (Turney, 2006b) used automatically generated patterns.”
A Uniform Approach to Analogies, Synonyms, Antonyms, and Associations (2008, methodology): “Turney et al. (2003) used a combination of 13 independent modules.”
Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn’t. (2016, background): “However, the current sets tend to only include a certain type of relations (semantic-only: SAT (Turney et al., 2003), SemEval2012-Task2 (Jurgens et al., 2012), morphology-only: MSR (Mikolov et al., 2013b)).”
Corpus-based Learning of Analogies and Semantic Relations (2005, methodology): “1, the VSM algorithm performs as well as an ensemble of twelve other modules (Turney et al., 2003).”
Computational Approaches to Sentence Completion (2012, methodology): “Among the narrower tasks, the identification of synonyms and antonyms has been studied by (Landauer and Dumais, 1997; Mohammed et al., 2008; Mohammed et al., 2011; Turney et al., 2003; Turney, 2008), who used questions from the Test of English as a Foreign Language (TOEFL), Graduate Record Exams (GR”
The (too Many) Problems of Analogical Reasoning with Word Vectors (2017, background): “In terms of practical NLP tasks, Turney et al. (2003) introduced the task of solving SAT 1 analogy problems by choosing from several provided options.”
Distributional Semantics Beyond Words: Supervised Learning of Analogy and Paraphrase (2013, methodology): “We experiment with three datasets, a collection of 374 ﬁve-choice questions from the SAT college entrance exam (Turney et al., 2003), a mod-iﬁed ten-choice variation of the SAT questions 4 Weka is available at http://www.cs.waikato.ac.nz/ml/weka/ . http://”
From distributional to semantic similarity (2004, methodology): “This problem is quite similar to the vocabulary tests used by Turney et al. (2003) for evaluation.”
Align, Disambiguate and Walk: A Unified Approach for Measuring Semantic Similarity (2013, unknown): “…(Bullinaria and Levy, 2007) 85.00% GLSA (Matveeva et al., 2005) 86.25% LSA (Rapp, 2003) 92.50% ADW Jac 93.75 ± 2.5% ADW WO 95.00% ADW Cos 96.25% PR (Turney et al., 2003) 97.50% PCCP (Bullinaria and Levy, 2012) 100.00% For the similarity judgment evaluation, we used as benchmark the RG-65 dataset…”
Extracting semantic representations from word co-occurrence statistics: stop-lists, stemming, and SVD (2012, background): “The details of exactly how that is done affect the results that emerge, but most reasonable approaches lead to around 97.5%, which still exceeds the previous best corpus-based result (Rapp, 2003) and equals the existing overall state-of-the-art result for the TOEFL task (Turney et al., 2003).”
Roget's thesaurus and semantic similarity (2012, methodology): “Bigham et al. (2003) use their similarity metric to answer analogy problems.”
Semantic Taxonomy Induction from Heterogenous Evidence (2006, background): “, 2003), a variety of verb relations (Chklovski and Pantel, 2004), and general purpose analogy relations (Turney et al., 2003).”
Polarity Inducing Latent Semantic Analysis (2012, unknown): “Combinations of approaches are studied in (Turney et al., 2003).”
Learning Syntactic Patterns for Automatic Hypernym Discovery (2004, background): “Thus a wide variety of recent research has focused on finding methods for automatically learning taxonomic relations and constructing semantic hierarchies [1, 2, 3, 4, 6, 8, 9, 10, 16, 18, 19, 20, 21, 22].”