Subtopic structuring for full-length document access

By Marti A. Hearst, C. Plaunt. 1993

We argue that the advent of large volumes of full-length text, as opposed to short texts like abstracts and newswire, should be accompanied by corresponding new approaches to information access. Toward this end, we discuss the merits of imposing structure on full-length text documents; that is, a partition of the text into coherent multi-paragraph units that represent the pattern of subtopics that comprise the text. Using this structure, we can make a distinction between the main topics, which occur throughout the length of the text, and the subtopics, which are of only limited extent. We discuss why recognition of subtopic structure is important and how, to some degree of accuracy, it can be found. We describe a new way of specifying queries on full-length documents and then describe an experiment in which making use of the recognition of local structure achieves better results on a typical information retrieval task than does a standard IR measure.

Subfields: Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Retrieval & Search

Citations in the corpus

Pretrained Transformers for Text Ranking: BERT and Beyond (2020, neutral): “ent with previous results reported in the information retrieval literature. There is a long thread of work, dating back to the 1990s, that leverages passage retrieval techniques for document ranking [Hearst and Plaunt, 1993, Callan, 1994, Wilkinson, 1994, Kaszkiel and Zobel, 1997, Clarke et al., 200”
Automatic Word Sense Discrimination (1998, background): “| An astronaut on a mission in space sends pictures back to Earth.”
A Critique and Improvement of an Evaluation Metric for Text Segmentation (2002, background): “Text segmentation issues are also important for passage retrieval, a subproblem of information retrieval (Hearst and Plaunt, 1993; Salton, Allan, and Buckley, 1993; Callan, 1994; Kaszkiel and Zobel, 1997).”
Test Collection Based Evaluation of Information Retrieval Systems (2010, background): “Initial work [117, 212] focused on using passage retrieval to improve document retrieval, which meant that existing document test collections could be used unaltered.”
Cross-Domain Modeling of Sentence-Level Evidence for Document Retrieval (2019, background): “This approach, in fact, is well motivated: There is a long thread of work in the information retrieval literature, dating back decades, that leverages passage retrieval techniques for document ranking (Hearst and Plaunt, 1993; Callan, 1994; Kaszkiel and Zo-bel, 1997; Clarke et al., 2000).”
Passage-level evidence in document retrieval (1994, background): “TextTiling [9]) might yield better results.”
Relevance ranking for one to three term queries (1997, background): “…& Zobel, 1997; Knaus, Mittendorf, SchaÈ uble, & Sheridan, 1995), most passage retrieval methods divide a document into passages at the time it is added to the database (Allan, 1995; Callan, 1994; Hearst & Plaunt, 1993; Salton, Allan, & Buckley, 1993; Wilkinson, 1994; Wilkinson & Zobel, 1994; ).”
Advances in domain independent linear text segmentation (2000, methodology): “The uses of this procedure include information retrieval (Hearst and Plaunt, 1993; Hearst, 1994; Yaari, 1997; Reynar, 1999), summarization (Reynar, 1998), text understanding, anaphora resolution (Kozima, 1993), language modelling (Morris and Hirst, 1991; Beeferman et al., 1997b) and improving…”
Is searching full text more effective than searching abstracts? (2009, methodology): “As a start, I have experimented with two different indexing granularities (full articles and spans), but alternative approaches include sliding windows [18,20], multi-para-graph segments [19], hierarchically-overlapping segments [38,39], and segments based on topic shifts [17].”
Information Retrieval: A Biomedical and Health Perspective (2020, background): “However, this process has been shown in some instances to lead to fraud, as those submitting papers conspire to list colleagues or even nonscientists as suggested peer reviewers [183, 184].”