S2ORC: The Semantic Scholar Open Research Corpus

By Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Michael Kinney, Daniel S. Weld. 2020

814 citations recorded by the source corpus.

We introduce S2ORC, a large corpus of 81.1M English-language academic papers spanning many academic disciplines. The corpus consists of rich metadata, paper abstracts, resolved bibliographic references, as well as structured full text for 8.1M open access papers. Full text is annotated with automatically-detected inline mentions of citations, figures, and tables, each linked to their corresponding paper objects. In S2ORC, we aggregate papers from hundreds of academic publishers and digital archives into a unified source, and create the largest publicly-available collection of machine-readable academic text to date. We hope this resource will facilitate research and development of tools and tasks for text mining over academic text.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, methodology): “…permission to use the data from specific providers within the context of BigScience whenever possible (such as for the AI26-managed S2ORC corpus of Lo et al. (2020) or articles from the French newspaper Le Monde7); (ii) we kept individual sources separate until the final stages of preprocessing…”
SCROLLS: Standardized CompaRison Over Long Language Sequences (2022, background): “Qasper (Dasigi et al., 2021): A question answering dataset over NLP papers ﬁltered from the Semantic Scholar Open Research Corpus (S2ORC) (Lo et al., 2020).”
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks (2020, positive): “22M full-text papers from S2ORC (Lo et al., 2020) 8.”
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023, background): “Non-trivial datasets overlap included s2orc (Lo et al., 2020), Arxiv (Clement et al., 2019) and the PubMed Central subset of the Pile (Gao et al., 2020).”
Continual Training of Language Models for Few-Shot Learning (2022, methodology): “Datasets: We use 4 unlabeled domain datasets : Yelp Restaurant (Xu et al., 2019), AI Papers (Lo et al., 2020), ACL Papers (Lo et al., 2020) and AG-News (Zhang et al., 2015) and their 4 corresponding end-task classiﬁcation datasets .”
ELLE: Efficient Lifelong Pre-training for Emerging Data (2022, background): “…scenario where streaming data from 5 domains is gathered sequentially, i.e., the concatenation of W IKIPEDIA and B OOK C ORPUS (WB) (Zhu et al., 2015), N EWS A RTICLES (N S ) (Zellers et al., 2019), A MA - ZON R EVIEWS (R EV ) (He and McAuley, 2016), B IOMEDICAL P APERS (B IO ) (Lo et al., 2020)”
DEMix Layers: Disentangling Domains for Modular Language Modeling (2021, neutral): “2M full-text medical papers from S2ORC (Lo et al., 2020) 9.”
A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers (2021, methodology): “To adhere to copyright, we have restricted ourselves to arXiv papers released under a CC-BY-* license, as identified via Unpaywall, which was used in the S2ORC (Lo et al., 2020) dataset construction.”
Towards General Text Embeddings with Multi-stage Contrastive Learning (2023, methodology): “We also include citation pairs from the S2ORC dataset (Lo et al., 2020).”
Knowledge Inheritance for Pre-trained Language Models (2021, positive): “Speciﬁcally, keeping data size the same, we mix Wikipedia and Book-Corpus (WB) used before with computer science (CS) papers from S2ORC (Lo et al., 2020), whose domain is distinct from WB, using different proportions, i.e., WB : CS = { 1 : 2 , 2 : 1 , 3 : 1 , 4 : 1 } , respectively.”
MSˆ2: Multi-Document Summarization of Medical Studies (2021, methodology): “NLP for scientific text has been gaining interest recently with work spanning the whole NLP pipeline: datasets (S2ORC (Lo et al., 2020), CORD-19 (Wang et al.”
SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore (2023, methodology): “We also collect full-text articles from the Semantic Scholar Research Corpus (Lo et al., 2020, S2ORC), either licensed as public domain or CC-BY.”
M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation (2024, neutral): “Particularly, the unsupervised data is curated by extracting the rich-semantic structures, e.g., title-body, title-abstract, instruction-output, etc., within a wide variety of multi-lingual corpora, including Wikipedia, S2ORC (Lo et al., 2020), xP3 (Muennighoff et al., 2023), mC4 (Raffel et al., 202”
AllenAct: A Framework for Embodied AI Research (2020, methodology): “This analysis used the S2ORC [32]. improvements on standard tasks and benchmarks, it is crucial to understand what components of systems matter most and which do not matter at all.”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, background): “Academic S2ORC [86] Large scale collection of more than 80M research papers written in English.”
Time Waits for No One! Analysis and Challenges of Temporal Misalignment (2021, background): “(Wadden et al., 2020; Lo et al., 2020); it is expected to show a great deal of variation over time as ideas and terminology evolve.”