Catching the Drift: Probabilistic Content Models, with Applications to Generation and Summarization

By Regina Barzilay, Lillian Lee. North American Chapter of the Association for Computational Linguistics, 2004

417 citations recorded by the source corpus.

We consider the problem of modeling the content structure of texts within a specic domain, in terms of the topics the texts address and the order in which these topics appear. We rst present an effective knowledge-lean method for learning content models from unannotated documents, utilizing a novel adaptation of algorithms for Hidden Markov Models. We then apply our method to two complementary tasks: information ordering and extractive summarization. Our experiments show that incorporating content models in these applications yields substantial improvement over previously-proposed methods.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/984efc8932edb635d09ec1a5fd8fc1d1ceccad45

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

What to talk about and how? Selective Generation using LSTMs with Coarse-to-Fine Alignment (2015, methodology): “With regards to the former, Barzilay and Lee (2004) model the content structure from unan-notated documents and apply it to the application of text summarization.”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, background): “Bateman and Zock (2005) cover the cognitive, social and computational dimensions of nlg. McDonald (2010) offers a general characterization of nlg as ‘the process by which thought is rendered into language’ (p. 121). Wanner (2010) zooms in on automatic generation of reports, while Di Eugenio and Gree”
Modeling Local Coherence: An Entity-Based Approach (2005, background): “Text structuring algorithms (Lapata, 2003; Barzilay and Lee, 2004; Karamanis et al., 2004) are commonly evaluated by their performance at information-ordering.”
Large-scale Analysis of Counseling Conversations: An Application of Natural Language Processing to Mental Health (2016, methodology): “Most approaches employ Hidden Markov Model-like models (Barzilay and Lee, 2004; Ritter et al., 2010; Paul, 2012; Yang et al., 2014) which are also used in this work to model progression through conversation stages.”
Generating Summaries with Topic Templates and Structured Convolutional Decoders (2019, background): “(2018) represent the target summaries as a single long sequence, despite the fact that documents are organized into topically coherent text segments, exhibiting a specific structure in terms of the content they discuss (Barzilay and Lee, 2004).”
Unsupervised Modeling of Twitter Conversations (2010, methodology): “Predicting post order on open-domain Twitter conversations is a much more difficult task than on topic-focused news data (Barzilay and Lee, 2004).”
BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization (2019, background): “While prior work uses probabilistic topic models (Barzilay and Lee, 2004; Haghighi and Vanderwende, 2009) or relies on classiﬁers trained with sophisticated features (Yang et al., 2017), we focus on salient words and their occurrences in the input.”
Probabilistic Frame Induction (2013, methodology): “Prior work has demonstrated that promoting topic coherence in natural-language discourse helps discourse modeling (Barzilay and Lee, 2004).”
Sentence Ordering using Recurrent Neural Networks (2016, methodology): “While approaches based on the content model of Barzilay and Lee (2004) attempt to explicitly capture topics by discovering clusters in sentences, we observe that the neural approach implicitly discovers such structure.”
Language modelling ’ s generative model : is it rational ? (2004, background): “But the joke in part depends on seeing how a quite long initial description of the novel can be encapsulated in a five-word one: given only the five word capsule, the full novel could be about pretty well anything.”
A Bottom-Up Approach to Sentence Ordering for Multi-Document Summarization (2006, background): “Grouping sentences by topics has shown to improve the readability of a summary (Barzilay et al., 2002; Barzilay and Lee, 2004).”
Knowledge Graph-Augmented Abstractive Summarization with Semantic-Driven Cloze Reward (2020, background): “Modeling topic transitions and recurrences enables the identification of notable content, thus benefiting summarization (Barzilay and Lee, 2004).”
Exploring Content Models for Multi-Document Summarization (2009, background): “Another strand of work (Barzilay and Lee, 2004; Daumé III and Marcu, 2006; Eisenstein and Barzilay, 2008), has explored the use of structured probabilistic topic models to represent document content.”
Automatically Generating Wikipedia Articles: A Structure-Aware Approach (2009, background): “There has been a sizable amount of research on structure induction ranging from linear segmentation (Hearst, 1994) to content modeling (Barzilay and Lee, 2004).”
Sometimes We Want Ungrammatical Translations (2021, background): “The idea to randomly shuffle linguistic elements to evaluate NLP model performance goes back fairly far (Barzilay and Lee, 2004; Barzilay and Lapata, 2008), and has even been used to determine which”
A Survey of Text Summarization Techniques (2012, methodology): “In this case, Hidden Markov Models (HMM) that capture “story flow”—what topics are discussed in what order in the domain— can be trained [5, 28].”
Automatic Text Summarization of Newswire: Lessons Learned from the Document Understanding Conference (2005, background): “Previous research (Barzilay & Lee 2004) has shown that content selection and ordering can be approached in a very principled way for scenario-type inputs.”
Story Comprehension for Predicting What Happens Next (2017, background): “ing topic drift within a domain (Barzilay and Lee, 2004; Fung and Ngai, 2006), co-occurrence of words (Lapata, 2003; Soricut and Marcu, 2006), syntactic patterns (Louis and Nenkova, 2012) and discourse relations (Pitler and Nenkova, 2008; Lin”