Is searching full text more effective than searching abstracts?

By Jimmy J. Lin. 2009

99 citations recorded by the source corpus.

BackgroundWith the growing availability of full-text articles online, scientists and other consumers of the life sciences literature now have the ability to go beyond searching bibliographic records (title, abstract, metadata) to directly access full-text content. Motivated by this emerging trend, I posed the following question: is searching full text more effective than searching abstracts? This question is answered by comparing text retrieval algorithms on MEDLINE® abstracts, full-text articles, and spans (paragraphs) within full-text articles using data from the TREC 2007 genomics track evaluation. Two retrieval models are examined: bm25 and the ranking algorithm implemented in the open-source Lucene search engine.ResultsExperiments show that treating an entire article as an indexing unit does not consistently yield higher effectiveness compared to abstract-only search. However, retrieval based on spans, or paragraphs-sized segments of full-text articles, consistently outperforms abstract-only search. Results suggest that highest overall effectiveness may be achieved by combining evidence from spans and full articles.ConclusionUsers searching full text are more likely to find relevant articles than searching only abstracts. This finding affirms the value of full text collections for text retrieval and provides a starting point for future work in exploring algorithms that take advantage of rapidly-growing digital archives. Experimental results also highlight the need to develop distributed text retrieval algorithms, since full-text articles are significantly longer than abstracts and may require the computational resources of multiple machines in a cluster. The MapReduce programming model provides a convenient framework for organizing such computations.

Subfields: Information Systems, Molecular Biology, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Molecular Biology

In Statistics and Probability

Citations in the corpus

LINNAEUS: A species name identification system for biomedical literature (2010, background): “Nevertheless, increasing evidence suggests that information retrieval is better on full-text articles since coverage of biomedical terms is higher relative to abstracts [62-66].”
SPECTER: Document-level Representation Learning using Citation-informed Transformers (2020, background): “There have been other efforts in document representation learning such as extensions of word vectors to documents (Le and Mikolov, 2014; Ganesh et al., 2016; Liu et al., 2017; Wu et al., 2018; Gysel et al., 2017), convolution-based methods (Liu et al., 2018; Zamani et al., 2018), and variational…”
SLEDGE: A Simple Yet Effective Baseline for Coronavirus Scientific Knowledge Search (2020, background): “Retrieval of scientiﬁc literature has been long-studied (Lawrence et al., 1999; Lalmas and Tombros, 2007; Hersh and Voorhees, 2009; Lin, 2008; Medlar et al., 2016; Sorkhei et al., 2017; Huang et al., 2019).”
Covidex: Neural Ranking Models and Keyword Search Infrastructure for the COVID-19 Open Research Dataset (2020, methodology): “Guided by previous work on searching full-text articles (Lin, 2009), we explored three separate indexing schemes:”