Measuring the Evolution of a Scientific Field through Citation Frames

By David Jurgens, Srijan Kumar, Raine Hoover, Daniel A. McFarland, Dan Jurafsky. 2018

308 citations recorded by the source corpus.

Citations have long been used to characterize the state of a scientific field and to identify influential works. However, writers use citations for different purposes, and this varied purpose influences uptake by future scholars. Unfortunately, our understanding of how scholars use and frame citations has been limited to small-scale manual citation analysis of individual papers. We perform the largest behavioral study of citations to date, analyzing how scientific works frame their contributions through different types of citations and how this framing affects the field as a whole. We introduce a new dataset of nearly 2,000 citations annotated for their function, and use it to develop a state-of-the-art classifier and label the papers of an entire field: Natural Language Processing. We then show how differences in framing affect scientific uptake and reveal the evolution of the publication venues and the field as a whole. We demonstrate that authors are sensitive to discourse structure and publication venue when citing, and that how a paper frames its work through citations is predictive of the citation count it will receive. Finally, we use changes in citation framing to show that the field of NLP is undergoing a significant increase in consensus.

Subfields: Artificial Intelligence, Molecular Biology, Statistics and Probability, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Statistics and Probability

In Natural Language Processing

Citations in the corpus

SciBERT: A Pretrained Language Model for Scientific Text (2019, background): “1https://github.com/google/ sentencepiece 2https://github.com/allenai/SciSpaCy ACL-ARC (Jurgens et al., 2018) and SciCite (Cohan et al., 2019) assign intent labels (e.g. Comparison, Extension, etc.) to sentences from scientific papers that cite other papers.”
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks (2020, neutral): “However, there are two tasks (SCIERC and ACL-ARC) in which ¬DAPT marginally improves performance over ROBERTA.”
Continual Training of Language Models for Few-Shot Learning (2022, neutral): “The other three corresponding end task datasets are SemEval-res (Xu et al., 2019), ACL-ARC (Jurgens et al., 2018), and SCIERC (Luan et al., 2018).”
ELLE: Efficient Lifelong Pre-training for Emerging Data (2022, methodology): “…we select a representative task that is relatively stable, i.e., MNLI (Williams et al., 2018), H YPER P ARTISAN (Kiesel et al., 2019), H ELPFULLNESS (McAuley et al., 2015), C HEM P ROT (Kringelum et al., 2016) and ACL-ARC (Jurgens et al., 2018) for WB, N S , R EV , B IO and CS, respectively.”
S2ORC: The Semantic Scholar Open Research Corpus (2020, background): “Other tasks that leverage citation contexts include classifying citation intent (Teufel et al., 2006; Jurgens et al., 2018; Cohan et al., 2019), identifying citation sentiment (Athar and Teufel, 2012), identifying meaningful citations (Valen-zuela et al., 2015), extracting key phrases (Caragea et…”
Continual Pre-training of Language Models (2023, background): “Their corresponding end-task classiﬁcation datasets are: 10 Restaurant 11 , Phone (Ding et al., 2008; Hu & Liu, 2004), Camera (Ding et al., 2008; Hu & Liu, 2004), ACL (ACL-ARC in (Jurgens et al., 2018)), AI (SCIERC in (Luan et al., 2018)), and PubMed (CHEMPORT in (Kringelum et al., 2016)).”
Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora (2021, methodology): “dataset (Dernoncourt and Lee, 2017) for the biomedical domain; ACL-ARC citation intent classification dataset (Jurgens et al., 2018) and SciERC relation extraction dataset (Luan et al.”
Merging Models with Fisher-Weighted Averaging (2021, methodology): “On the CS domain, we used the citation intent task of ACL-ARC [25] and the relation classiﬁcation task of S CI ERC [35].”
SciBERT: Pretrained Contextualized Embeddings for Scientific Text (2019, unknown): “Except for ACL-ARC and ScienceCite, all SCIBERT results are statistically significantly higher than BERTBase based on 95% bootstrap confidence intervals.”
GORC: A large contextual citation graph of academic papers (2019, methodology): “Contextual citation graphs have also been used for bibliometric analysis (Asatani et al., 2018; Jurgens et al., 2018; Tru-jillo and Long, 2018; Jurgens et al., 2016; Ding et al., 2014).”
The Diversity–Innovation Paradox in Science (2019, methodology): “ sign 29 of innovation. Here, we extend prior work by using recombinations of concepts used in scientific text, thus likely maintaining references to the explicit meaning of said concept combinations (6). Keywords are an alternative to citations. They constitute “plausible building blocks of content”
OAG-BERT: Pre-train Heterogeneous Entity-augmented Academic Language Models (2021, background): “Besides pre-training for general purposes, more and more language models are targeting at specific domains, such as BioBERT [26] for biomedical field and SciBERT [2] for academic field, which establish new state-of-the-art on many domain-related benchmarks such as named entity recognition [10, 32], ”
Should We Be Pre-training? An Argument for End-task Aware Training as an Alternative (2021, methodology): “Under the CS domain, we consider two tasks: ACL-ARC (Jurgens et al., 2018) and SCIERC (Luan et al.”
Black-box Prompt Learning for Pre-trained Language Models (2022, background): “, 2009), and four domain-specific datasets: CITATIONINTENT (Jurgens et al., 2018), SCIERC (Luan et al.”
ResearchGPT: Benchmarking and Training LLMs for End-to-End Computer Science Research Workflows (2025, unknown): “Early benchmarks in NLP were often built around relatively simple and domain-specific tasks, such as corpus annotation, sentiment classification, relation extraction, or fact-based question answering (Kim et al., 2003; Li et al., 2016; Luan et al., 2018; Jurgens et al., 2018; Jin et al., 2019).”