GreaseLM: Graph REASoning Enhanced Language Models for Question Answering
By Xikun Zhang, Antoine Bosselut, Michihiro Yasunaga, Hongyu Ren, Percy Liang, Christopher D. Manning, Jure Leskovec. International Conference on Learning Representations, 2022
306 citations recorded by the source corpus.
Answering complex questions about textual narratives requires reasoning over both stated context and the world knowledge that underlies it. However, pretrained language models (LM), the foundation of most modern QA systems, do not robustly represent latent relationships between concepts, which is necessary for reasoning. While knowledge graphs (KG) are often used to augment LMs with structured representations of world knowledge, it remains an open question how to effectively fuse and reason over the KG representations and the language context, which provides situational constraints and nuances. In this work, we propose GreaseLM, a new model that fuses encoded representations from pretrained LMs and graph neural networks over multiple layers of modality interaction operations. Information from both modalities propagates to the other, allowing language context representations to be grounded by structured world knowledge, and allowing linguistic nuances (e.g., negation, hedging) in the context to inform the graph representations of knowledge. Our results on three benchmarks in the commonsense reasoning (i.e., CommonsenseQA, OpenbookQA) and medical question answering (i.e., MedQA-USMLE) domains demonstrate that GreaseLM can more reliably answer questions that require reasoning over both situational constraints and structured knowledge, even outperforming models 8x larger.
Subfields: Artificial Intelligence, Molecular Biology
View Paper: https://www.semanticscholar.org/paper/4ab41d9780f1d1ac34d39fa7e527e73652507fcc
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Molecular Biology
Citations in the corpus
- Can Language Models Solve Graph Problems in Natural Language? (2023, background): “In multi-hop question answering or knowledge probing, LLMs implicitly find connections and paths among a vast network of entities and concepts [Creswell et al., 2023, Yu et al., 2022, Zhang et al., 2022a, He et al., 2021].”
- Deep Bidirectional Language-Knowledge Graph Pretraining (2022, unknown): “We also compare with existing KG-augmented QA models, QAGNN (Yasunaga et al. 2021) and GreaseLM (Zhang et al. 2022), which finetune a vanilla LM (i.e. RoBERTa-Large) with a KG on downstream tasks, but do not pretrain with a KG. GreaseLM is the existing top-performing model in this paradigm.”
- Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning (2023, methodology): “QA-GNN (Yasunaga et al., 2021) and Greaselm (Zhang et al., 2021) further adopt the graph neural network to capture the graph structure for reasoning.”
- Large Language Models on Graphs: A Comprehensive Survey (2023, methodology): “…MHA to take the structure tokens into consideration: where head u ( H With the asymmetric MHA mechanism, the node encoding process of the ( l + 1) -th layer will be: Along this line of work, GreaseLM [67] proposes to have a language encoding component and a graph encoding component in each layer.”
- Unifying Large Language Models and Knowledge Graphs: A Roadmap (2023, background): “…E + D + ED Entity/RelationExtractor DEKCOR[175] 2021 E AnswerReasoner DRLK[176] 2022 E AnswerReasoner OreoLM[177] 2022 E AnswerReasoner GreaseLM[178] 2022 E AnswerReasoner ReLMKG[179] 2022 E AnswerReasoner UniKGQA[43] 2023 E AnswerReasoner E :Encoder-onlyLLMs, D :Decoder-onlyLLMs, ED…”
- Retrieval And Structuring Augmented Generation with Large Language Models (2025, unknown): “Structure-enhanced LLM generation grounds model responses in explicit knowledge structures, reducing hallucinations and improving factual consistency.”