PyTorch-BigGraph: A Large-scale Graph Embedding System

By Adam Lerer, Ledell Yu Wu, Jiajun Shen, Timothée Lacroix, Luca Wehrstedt, Abhijit Bose, Alex Peysakhovich. USENIX workshop on Tackling computer systems problems with machine learning techniques, 2019

414 citations recorded by the source corpus.

Graph embedding methods produce unsupervised node features from graphs that can then be used for a variety of machine learning tasks. Modern graphs, particularly in industrial applications, contain billions of nodes and trillions of edges, which exceeds the capability of existing embedding systems. We present PyTorch-BigGraph (PBG), an embedding system that incorporates several modifications to traditional multi-relation embedding systems that allow it to scale to graphs with billions of nodes and trillions of edges. PBG uses graph partitioning to train arbitrarily large embeddings on either a single machine or in a distributed environment. We demonstrate comparable performance with existing embedding systems on common benchmarks, while allowing for scaling to arbitrarily large graphs and parallelization on multiple machines. We train and evaluate embeddings on several large social network graphs as well as the full Freebase dataset, which contains over 100 million nodes and 2 billion edges.

Subfields: Artificial Intelligence, Computer Networks and Communications, Statistical and Nonlinear Physics

View Paper: https://www.semanticscholar.org/paper/7ac58400e5063bed9b7c35f87e44ddb917ccf357

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Statistical and Nonlinear Physics

Citations in the corpus

Open Graph Benchmark: Datasets for Machine Learning on Graphs (2020, background): “A promising approach to improve scalability is to distribute training across multiple commodity GPUs (Zheng et al., 2020; Zhu et al., 2019; Lerer et al., 2019).”
Machine Learning on Graphs: A Model and Comprehensive Taxonomy (2020, background): “Methods that scale for large graphs [69, 119] require a Distributed Systems setup with many machines, such as MapReduce [31].”
Finding and Characterizing Information Warfare Campaigns (2020, methodology): “In addition to using the HOPE algorithm, we also tested Facebook’s Pytorch-Biggraph toolbox [153].”
Measuring Social Bias in Knowledge Graph Embeddings (2019, methodology): “All training is implemented using the PyTorch-BigGraph library [Lerer et al., 2019].”
Federated User Representation Learning (2019, background): “…a simple, communication-efﬁcient, scalable, privacy-preserving scheme, called FURL, to extend existing neural-network personalization to FL. FURL can personalize models in FL by learning task-speciﬁc user representations (i.e., embeddings) [15, 8, 23, 14, 11] or by personalizing model weights [29].”
Evaluating the Impact of Knowledge Graph Context on Entity Disambiguation Models (2020, methodology): “h burgeoning reuse through fine-tuning for several NLP tasks incl. NED [8]. On the other hand, KGs are increasingly being seen as a source of additional knowledge for Neural Networks. For instance in [4] released an embedding library for the Wikidata KG while the work by [9] introduced an extension ”
HyGCN: A GCN Accelerator with Hybrid Architecture (2020, background): “Many companies, such as Google [15], Facebook [7], and Alibaba [1] have deployed GCNs in data centers, which reflects the increasing importance and scope of upcoming applications.”
Zero-shot Entity Linking with Dense Entity Retrieval (2019, methodology): “For each training pair ( m i , e i ) in a batch of B pairs, the loss is computed as: Following previous work (e.g. (Lerer et al., 2019), (Humeau et al., 2019)), in training we consider the other elements of the batch as negatives.”
Contrastive Learning Using Graph Embeddings for Domain Adaptation of Language Models in the Process Industry (2025, unknown): “Following SciNCL, we employ PyTorch BigGraph (PBG) (Lerer et al., 2019) nodes are positioned closer together in the vector space while unconnected nodes are pushed farther apart.”