SPECTER: Document-level Representation Learning using Citation-informed Transformers

By Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, Daniel S. Weld. Annual Meeting of the Association for Computational Linguistics, 2020

853 citations recorded by the source corpus.

Representation learning is a critical ingredient for natural language processing systems. Recent Transformer language models like BERT learn powerful textual representations, but these models are targeted towards token- and sentence-level training objectives and do not leverage information on inter-document relatedness, which limits their document-level representation power. For applications on scientific documents, such as classification and recommendation, accurate embeddings of documents are a necessity. We propose SPECTER, a new method to generate document-level embedding of scientific papers based on pretraining a Transformer language model on a powerful signal of document-level relatedness: the citation graph. Unlike existing pretrained language models, Specter can be easily applied to downstream applications without task-specific fine-tuning. Additionally, to encourage further research on document-level models, we introduce SciDocs, a new evaluation benchmark consisting of seven document-level tasks ranging from citation prediction, to document classification and recommendation. We show that Specter outperforms a variety of competitive baselines on the benchmark.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/a3e4ceb42cbcd2c807d53aff90a8cb1f5ee3f031

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Percy Liang
2. Luke S. Zettlemoyer
3. Mohit Bansal
4. Noah A. Smith
5. Hannaneh Hajishirzi
6. Tie-Yan Liu
7. Graham Neubig
8. Jianfeng Gao

Citations in the corpus

• Benchmarking Retrieval-Augmented Generation for Medicine (2024, methodology): “…in M ED R AG due to limited resources, including a lexical retriever (BM25, Robertson et al., 2009), a general-domain semantic retriever (Contriever, Izacard et al., 2022), a scientific-domain retriever (SPECTER, Cohan et al., 2020), and a biomedical-domain retriever (MedCPT, Jin et al., 2023a).”
• LinkBERT: Pretraining Language Models with Document Links (2022, methodology): “Citation links are often used to improve summarization and recommendation of academic papers (Qazvinian & Radev, 2008; Yasunaga et al., 2019; Bhagavatula et al., 2018; Khadka et al., 2020; Cohan et al., 2020).”
• BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models (2021, neutral): “68 Citation-Prediction Scientific SCIDOCS [9] 3 Binary —- —- 1,000 25,657 4.”
• Large Language Models on Graphs: A Comprehensive Survey (2023, methodology): “SPECTER [51] constructs the positive text/node pairs with the citation relation, explores random negatives and structure hard negatives, and fine-tunes SciBERT [25] with the triplet loss.”
• BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining (2022, background): “…[16], BioASQ [30, 31] Document Classification BERT [2], BlueBERT [8], SciBERT [18], SPECTER [32], PubMedBERT [9], BioELECTRA [28], LinkBERT [29] HoC [17], SciDocs [32] Document Classification Document classification is to classify a document into predefined categories (single label or multi label).”
• ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction (2021, background): “, 2018), Quora, SCIDOCS (Cohan et al., 2020), and SciFact (Wadden et al.”
• CORD-19: The Covid-19 Open Research Dataset (2020, methodology): “21 SPECTER (Cohan et al., 2020) paper embeddings computed using paper titles and abstracts are being released with each CORD-19 update.”
• MTEB: Massive Text Embedding Benchmark (2022, methodology): “Scientific datasets, such as SciDocsRR, SciFact, ArxivClustering, show high similarities among each other even when coming from different tasks (Reranking, Retrieval and Clustering in this case).”
• Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, neutral): “Other tasks Wikipedia FEVER [66] – 140,085 6,666 6,666 5,416,568 Wikipedia Climate-FEVER [67] – – – 1,535 5,416,593 Scitific SciFact [68] 809 920 – 300 5,183 Scitific SciDocs [69] – – – 1,000 25,657”
• One Embedder, Any Task: Instruction-Finetuned Text Embeddings (2022, neutral): “…agnews (Zhang et al., 2015) Asymmetric 45,000 AllNLI (Bowman et al., 2015) Symmetric 50,000 npr (Li, 2020) Asymmetric 25,000 specter_train_triples (Cohan et al., 2020) Symmetric 50,000 ccnews_title_text (Hamborg et al., 2017) Asymmetric 25,000 triviaqa (Joshi et al., 2017) Asymmetric 50,000…”
• Task-aware Retrieval with Instructions (2022, neutral): “, 2017), SciDocs (Cohan et al., 2020), Climate- Fever (Diggelmann et al.”
• MSˆ2: Multi-Document Summarization of Medical Studies (2021, methodology): “We compute SPECTER paper embeddings (Cohan et al., 2020) using the title and”