Skip to content

Latest commit

 

History

History
585 lines (512 loc) · 68.5 KB

README.md

File metadata and controls

585 lines (512 loc) · 68.5 KB
Raw

Awesome Graph Self-Supervised Learning

PRs WelcomeAwesomeGitHub stars GitHub forks

A curated list for awesome self-supervised graph representation learning resources. Inspired by awesome-deep-vision, awesome-adversarial-machine-learning, awesome-deep-learning-papers, awesome-architecture-search, and awesome-self-supervised-learning.

Why Self-Supervised?

Self-Supervised Learning has become an exciting direction in AI community.

  • Jitendra Malik: "Supervision is the opium of the AI researcher"
  • Alyosha Efros: "The AI revolution will not be supervised"
  • Yann LeCun: "self-supervised learning is the cake, supervised learning is the icing on the cake, reinforcement learning is the cherry on the cake"

Table of Contents

Overview

We extend the concept of self-supervised learning, which first emerged in the fields of computer vision and natural language processing, to present a timely and comprehensive review of the existing SSL techniques for graph data. Specifically, we divide existing graph SSL methods into three categories: contrastive, generative, and predictive as shown below.

  • Contrastive Learning: it contrasts the views generated by different data augmentation methods. The information about the differences and sameness between data-data pairs (inter-data) is used as self-supervision signals.
  • Generative Learning: it focuses on the (intra-data) information embedded in the data, generally based on prtext tasks such as reconstruction, which exploit the attributes and structure of the data itself as self-supervision signals.
  • Predictive Learning: it generally self-generates labels from graph data through some simple statistical analysis, or expert knowledge, and designs prediction-based pretext tasks based on the self-generated labels to handle the data-label relationship.

Training Strategy

Considering the relationship among bottleneck encoders, self-supervised pretext tasks, and downstream tasks, the training strategies can be divided into three categories: Pre-training and Fine-tuning (P&F), Joint Learning (JL), and Unsupervised Representation Learning (URL), with their detailed workflow shown below.

  • Pre-train&Fine-tune (P&F): it first pre-trains the encoder with unlabeled nodes by the self-supervised pretext tasks. The pre-trained encoder’s parameters are then used as the initialization of the encoder used in supervised fine-tuning for downstream tasks.
  • Joint Learning (JL): an auxiliary pretext task with self-supervision is included to help learn the supervised downstream task. The encoder is trained through both the pretext task and the downstream task simultaneously.
  • Unsupervised Representation Learning (URL): it first pre-trains the encoder with unlabeled nodes by the self-supervised pretext tasks. The pre-trained encoder’s parameters are then frozen and used in the supervised downstream task with additional labels.

Contrastive Learning

A general framework for contrastive learning is shown below. The two contrasting components may be local, contextual, or global, corresponding to node-level (marked in red), subgraph-level (marked in green), or graph-level (marked in yellow) information in the graph. The contrastive learning can thus contrast two views (at the same or different scales), which leads to two categories of algorithm: (1) same-scale contrasting, including Local-Local (L-L) contrasting, Context-Context (C-C) contrasting, and Global-Global (G-G) contrasting; and (2) cross-scale contrasting, including Local-Context (L-C) contrasting, Local-Global (L-G) contrasting, and Context-Global (C-G) contrasting.

Global-Global Contrasting

  • GraphCL: Graph Contrastive Learning with Augmentations.
    • Y. You, T. Chen, Y. Sui, T. Chen, Z. Wang, and Y. Shen. NIPS 2020. [pdf] [code]
  • IGSD: Iterative Graph Self-Distillation.
    • H. Zhang, S. Lin, W. Liu, P. Zhou, J. Tang, X. Arxiv 2020. [pdf]
  • DACL: Towards Domain-Agnostic Contrastive Learning.
    • V. Verma, M.-T. Luong, K. Kawaguchi, H. Pham, andQ. V. Le. Arxiv 2020. [pdf]
  • LCC: Label Contrastive Coding Based Graph Neural Network for Graph Classification.
    • Y. Ren, J. Bai, and J. Zhang. Arxiv 2021. [pdf] [code]
  • CCGL: Contrastive Cascade Graph Learning.
    • X. Xu, F. Zhou, K. Zhang, and S. Liu. TKDE 2022. [pdf] [code]
  • CSSL: Contrastive Self-Supervised Learning for Graph Classification.
    • J. Zeng and P. Xie. Arxiv 2020. [pdf]

Context-Context Contrasting

  • GCC: Graph Contrastive Coding for Graph Neural Network Pre-training.
    • J. Qiu, Q. Chen, Y. Dong, J. Zhang, H. Yang, M. Ding, K. Wang, and J. Tang. KDD 2020. [pdf] [code]

Local-Local Contrasting

  • CDNMF: Contrastive Deep Nonnegative Matrix Factorization for Community Detection.
    • Y. Li, J. Chen, C. Chen, L. Yang, Z. Zheng. ICASSP 2024. [pdf] [code]
  • GRACE: Deep Graph Contrastive Representation Learning.
    • Y. Zhu, Y. Xu, F. Yu, Q. Liu, S. Wu, and L. Wang. Arxiv 2020. [pdf] [code]
  • GCA: Graph Contrastive Learning with Adaptive Augmentation.
    • Y. Zhu, Y. Xu, F. Yu, Q. Liu, S. Wu, and L. Wang. Arxiv 2020. [pdf] [code]
  • GROC: Towards Robust Graph Contrastive Learning.
    • N. Jovanovi´c, Z. Meng, L. Faber, and R. Wattenhofer. Arxiv 2021. [pdf]
  • SEPT: Socially-Aware Self-Supervised Tri-Training for Recommendation.
    • J. Yu, H. Yin, M. Gao, X. Xia, X. Zhang, and N. Q. V.Hung. Arxiv 2021. [pdf] [code]
  • STDGI: Spatio-Temporal Deep Graph Infomax.
    • F. L. Opolka, A. Solomon, C. Cangea, P. Veliˇckovi´c, P. Li` o, and R. D. Hjelm. Arxiv 2019. [pdf]
  • GMI: Graph Representation Learning via Graphical Mutual Information Maximization.
    • L. Yu, S. Pei, C. Zhang, L. Ding, J. Zhou, L. Li, and X. Zhang. WWW 2020. [pdf] [code]
  • KS2L: Self-Supervised Smoothing Graph Neural Networks.
    • L. Yu, S. Pei, C. Zhang, L. Ding, J. Zhou, L. Li, and X. Zhang. Arxiv 2020. [pdf]
  • CG3: Contrastive and Generative Graph Convolutional Networks for Graph-based Semi-Supervised Learning.
    • S. Wan, S. Pan, J. Yang, and C. Gong. Arxiv 2020. [pdf]
  • BGRL: Bootstrapped Representation Learning on Graphs.
    • S. Thakoor, C. Tallec, M. G. Azar, R. Munos, P. Veliˇckovi´c, and M. Valko. Arxiv 2021. [pdf][code]
  • SelfGNN: Self-supervised Graph Neural Networks without Explicit Negative Sampling.
    • Z. T. Kefato and S. Girdzijauskas. Arxiv 2021. [pdf] [code]
  • HeCo: Self-supervised Heterogeneous Graph Neural Network with Co-contrastive Learning.
    • X. Wang, N. Liu, H. Han, and C. Shi. Arxiv 2021. [pdf] [code]
  • PT-DGNN: Pre-training on Dynamic Graph Neural Networks.
    • J. Zhang, K. Chen, and Y. Wang. Arxiv 2021. [pdf] [code]
  • COAD: Coad: Contrastive Pretraining with Adversarial Fine-tuning for Zero-shot Expert Linking.
    • B. Chen, J. Zhang, X. Zhang, X. Tang, L. Cai, H. Chen, C. Li, P. Zhang, and J. Tang. Arxiv 2020. [pdf] [code]
  • Contrast-Reg: Improving Graph Representation Learning by Contrastive Regularization.
    • K. Ma, H. Yang, H. Yang, T. Jin, P. Chen, Y. Chen, B. F. Kamhoua, and J. Cheng. Arxiv 2021. [pdf]
  • C-SWM: Contrastive Learning of Structured World Models.
    • T. Kipf, E. van der Pol, and M. Welling. *Arxiv 2019. [pdf] [code]

Local-Global Contrasting

  • DGI: Deep Graph Infomax.
    • P. Velickovic, W. Fedus, W. L. Hamilton, P. Li` o, Y. Bengio, and R. D. Hjelm. ICLR 2019. [pdf] [code]
  • HDMI: Hdmi: High-order Deep Multiplex Infomax.
    • B. Jing, C. Park, and H. Tong. Arxiv 2021. [pdf]
  • DMGI: Unsupervised Attributed Multiplex Network Embedding.
    • C. Park, D. Kim, J. Han, and H. Yu. AAAI 2020. [pdf] [code]
  • MVGRL: Contrastive Multi-View Representation Learning on Graphs.
    • K. Hassani and A. H. K. Ahmadi. ICML 2020. [pdf] [code]
  • HDGI: Heterogeneous Deep Graph Infomax.
    • Y. Ren, B. Liu, C. Huang, P. Dai, L. Bo, and J. Zhang. Arxiv 2019. [pdf] [code]

Local-Context Contrasting

  • CDNMF: Contrastive Deep Nonnegative Matrix Factorization for Community Detection.
    • Y. Li, J. Chen, C. Chen, L. Yang, Z. Zheng. ICASSP 2024. [pdf] [code]
  • Subg-Con: Sub-graph Contrast for Scalable Self-Supervised Graph Representation Learning.
    • Y. Jiao, Y. Xiong, J. Zhang, Y. Zhang, T. Zhang, and Y. Zhu. Arxiv 2020. [pdf] [code]
  • Cotext Prediction: Strategies for Pre-training Graph Neural Networks.
    • W. Hu, B. Liu, J. Gomes, M. Zitnik, P. Liang, V. S. Pande, and J. Leskovec. ICLR 2020. [pdf] [code]
  • GIC: Leveraging Cluster-level Node Information for Unsupervised Graph Representation Learning.
    • C. Mavromatis and G. Karypis. Arxiv 2020. [pdf] [code]
  • GraphLoG: Self-Supervised Graph-level Representation Learning with Local and Global Structure.
    • M. Xu, H. Wang, B. Ni, H. Guo, and J. Tang. OpenReview 2021. [pdf] [code]
  • MHCN: Self-Supervised Multi-channel Hypergraph Convolutional Network for Social Recommendation.
    • J. Yu, H. Yin, J. Li, Q. Wang, N. Q. V. Hung, and X. Zhang. Arxiv 2021. [pdf] [code]
  • EGI: Transfer Learning of Graph Neural Networks with Ego-graph Information Maximization.
    • Q. Zhu, Y. Xu, H.Wang, C. Zhang, J. Han, and C. Yang. Arxiv 2020. [pdf] [code]

Context-Global Contrasting

  • MICRO-Graph: Motif-Driven Contrastive Learning of Graph Representations.
    • S. Zhang, Z. Hu, A. Subramonian, and Y. Sun. Arxiv 2020. [pdf] [code]
  • InfoGraph: Unsupervised and Semi-Supervised Graph-level Representation Learning via Mutual Information Maximization.
    • F. Sun, J. Hoffmann, V. Verma, and J. Tang. ICLR 2020. [pdf] [code]
  • SUGAR: Subgraph Neural Network with Reinforcement Pooling and Self-Supervised Mutual Information Mechanism.
    • Q. Sun, H. Peng, J. Li, J. Wu, Y. Ning, P. S. Yu, and L. He. Arxiv 2021. [pdf] [code]
  • BiGI: Bipartite Graph Embedding via Mutual Information Maximization.
    • J. Cao, X. Lin, S. Guo, L. Liu, T. Liu, and B. Wang. WSDM 2021. [pdf] [code]
  • HTC: Graph Representation Learning by Ensemble Aggregating Subgraphs via Mutual Information Maximization.
    • C. Wang and Z. Liu. Arxiv 2021. [pdf]
  • DITNet: Drug Target Prediction using Graph Representation Learning via Substructures Contrast.
    • S. Cheng, L. Zhang, B. Jin, Q. Zhang, and X. Lu. Preprints 2021. [pdf] [code]

Generative Learning

Graph Autoencoding

  • CDNMF: Contrastive Deep Nonnegative Matrix Factorization for Community Detection.
    • Y. Li, J. Chen, C. Chen, L. Yang, Z. Zheng. ICASSP 2024. [pdf] [code]
  • GraphMAE: Self-supervised Masked Graph Autoencoders
    • Z. Hou, X. Liu, Y. Cen, Y. Dong, H. Yang, C. Wang, and J. Tang. KDD 2022 [pdf] [code]
  • Graph Completion: When Does Self-Supervision Help Graph Convolutional Networks?
    • Y. You, T. Chen, Z. Wang, and Y. Shen. PMLR 2020. [pdf] [code]
  • Node Attribute Masking: Self-Supervised Learning on Graphs: Deep Insights and New Direction.
    • W. Jin, T. Derr, H. Liu, Y. Wang, S. Wang, Z. Liu, and J. Tang. Arxiv 2020. [pdf] [code]
  • Edge Attribute Masking: Strategies for Pre-training Graph Neural Networks.
    • W. Hu, B. Liu, J. Gomes, M. Zitnik, P. Liang, V. S. Pande, and J. Leskovec. ICLR 2020. [pdf] [code]
  • Node Attribute and Embedding Denoising: Graph-based Neural Network Models with Multiple Self-Supervised Auxiliary Tasks.
    • F. Manessi and A. Rozza. Arxiv 2020. [pdf]
  • Adjacency Matrix Reconstruction: Self-Supervised Training of Graph Convolutional Networks.
    • Q. Zhu, B. Du, and P. Yan. Arxiv 2020. [pdf]
  • Graph Bert: Only Attention is Needed for Learning Graph Representations.
    • J. Zhang, H. Zhang, C. Xia, and L. Sun. Arxiv 2020. [pdf] [code]
  • Pretrain-Recsys: Pretraining Graph Neural Networks for Cold-start Users and Items Representation.
    • B. Hao, J. Zhang, H. Yin, C. Li, and H. Chen. WSDM 2021. [pdf] [code]
  • SLAPS: Self-Supervision Improves Structure Learning for Graph Neural Networks.
    • B. Fatemi, L. E. Asri, and S. M. Kazemi. Arxiv 2021. [pdf] [code]
  • G-BERT: Pre-Training of Graph Augmented Transformers for Medication Recommendation.
    • J. Shang, T. Ma, C. Xiao, and J. Sun. Arxiv 2019. [pdf] [code]

Graph Autoregression

  • GPT-GNN: Generative Pre-training of Graph Neural Networks.
    • Z. Hu, Y. Dong, K. Wang, K. Chang, and Y. Sun. KDD 2020. [pdf] [code]

Predictive Learning

A comparison of the predictive learning is shown below. The predictive method generally self-generates labels from graph data and then designs prediction-based pretext tasks based on the self-generated labels. Categorized by how the labels areobtained, we summarize predictive learning methods forgraph data into four categories:

  • Node Property Prediction: it pre-calculates the node properties, such as node degree and used them as self-supervised labels.
  • Context-based Prediction: the local or global contextual information in the graph, such as the shortest path length between nodes can be extracted as labels to help with self-supervised learning.
  • Self-Training: it applies algorithms such as unsupervised clustering to obtain pseudo-labels and then updates the pseudo-label set of the previous stage based on the prediction results or losses.
  • Domain Knowledge-based Prediction: the domain knowledge, such as expert knowledge or specialized tools, can be used in advance to obtain informative labels.

Node Property Prediction

  • Node Property Prediction: Self-Supervised Learning on Graphs: Deep Insights and New Direction.
    • W. Jin, T. Derr, H. Liu, Y. Wang, S. Wang, Z. Liu, and J. Tang. Arxiv 2020. [pdf] [code]

Context-based Prediction

  • S2GRL: Self-Supervised Graph Representation Learning via Global Context Prediction.
    • Z. Peng, Y. Dong, M. Luo, X.-M. Wu, and Q. Zheng. Arxiv 2020. [pdf]
  • PairwiseDistance: Self-Supervised Learning on Graphs: Deep Insights and New Direction.
    • W. Jin, T. Derr, H. Liu, Y. Wang, S. Wang, Z. Liu, and J. Tang. Arxiv 2020. [pdf] [code]
  • PairwiseAttsim: Self-Supervised Learning on Graphs: Deep Insights and New Direction.
    • W. Jin, T. Derr, H. Liu, Y. Wang, S. Wang, Z. Liu, and J. Tang. Arxiv 2020. [pdf] [code]
  • Distance2Cluster: Self-Supervised Learning on Graphs: Deep Insights and New Direction.
    • W. Jin, T. Derr, H. Liu, Y. Wang, S. Wang, Z. Liu, and J. Tang. Arxiv 2020. [pdf] [code]
  • EdgeMask: Self-Supervised Learning on Graphs: Deep Insights and New Direction.
    • W. Jin, T. Derr, H. Liu, Y. Wang, S. Wang, Z. Liu, and J. Tang. Arxiv 2020. [pdf] [code]
  • TopoTER: Unsupervised Learning of Topology Transformation Equivariant Representations.
    • X. Gao, W. Hu, and G.-J. Qi. OpenReview 2021. [pdf]
  • Centrality Score Ranking: Pretraining Graph Neural Networks for Generic Structural Feature Extraction.
    • Z. Hu, C. Fan, T. Chen, K.-W. Chang, and Y. Sun. Arxiv 2019. [pdf]
  • Meta-path prediction: Self-supervised Auxiliary Learning with Meta-paths for Heterogeneous Graphs.
    • D. Hwang, J. Park, S. Kwon, K. Kim, J. Ha, and H. J. Kim. NIPS 2020. [pdf] [code]
  • SLiCE: Self-Supervised Learning of Contextual Embeddings for Link Prediction in Heterogeneous Networks.
    • P. Wang, K. Agarwal, C. Ham, S. Choudhury, and C. K. Reddy. Arxiv 2020. [pdf] [code]
  • Distance2Labeled: Self-Supervised Learning on Graphs: Deep Insights and New Direction.
    • W. Jin, T. Derr, H. Liu, Y. Wang, S. Wang, Z. Liu, and J. Tang. Arxiv 2020. [pdf] [code]
  • Distance2Labeled: Self-Supervised Learning on Graphs: Deep Insights and New Direction.
    • W. Jin, T. Derr, H. Liu, Y. Wang, S. Wang, Z. Liu, and J. Tang. Arxiv 2020. [pdf] [code]
  • HTM: Hop-count based Self-Supervised Anomaly Detection on Attributed Networks.
    • T. Huang, Y. Pei, V. Menkovski, and M. Pechenizkiy. Arxiv 2021. [pdf]

Self-Training

  • Multi-stage Self-training: Deeper insights into Graph Convolutional Networks for Semi-Supervised Learning.
  • Node Clustering and Partitioning: When Does Self-Supervision Help Graph Convolutional Networks.
    • Y. You, T. Chen, Z. Wang, and Y. Shen. PMLR 2020. [pdf] [code]
  • CAGAN: Cluster-Aware Graph Neural Networks for Unsupervised Graph Representation Learning.
    • Y. Zhu, Y. Xu, F. Yu, S. Wu, and L. Wang. Arxiv 2020. [pdf]
  • M3S: Multi-stage Self-Supervised Learning for Graph Convolutional Networks on Graphs with Few Labeled Nodes.
  • Cluster Preserving: Pretraining Graph Neural Networks for Generic Structural Feature Extraction.
    • Z. Hu, C. Fan, T. Chen, K.-W. Chang, and Y. Sun. Arxiv 2019. [pdf]
  • SEF: Self-Supervised Edge Features for Improved Graph Neural Network Training.
    • A. Sehanobish, N. G. Ravindra, and D. van Dijk. Arxiv 2020. [pdf][code]

Domain Knowledge-based Prediction

  • Contextual Molecular Property Prediction: Self-Supervised Graph Transformer on Large-Scale Molecular Data.
    • Y. Rong, Y. Bian, T. Xu, W. Xie, Y. Wei, W. Huang, and J. Huang. NIPS 2020. [pdf] [code]
  • Graph-level Motif Prediction: Self-Supervised Graph Transformer on Large-scale Molecular Data.
    • Y. Rong, Y. Bian, T. Xu, W. Xie, Y. Wei, W. Huang, and J. Huang. NIPS 2020. [pdf] [code]
  • DrRepair: Graph-based, Self-Supervised Program Repair from Diagnostic Feedback.

A summary of all the surveyed works is presented below.

A Summary of Methodology Details

About Graph Property, Pretext Task, Data Augmentation, Objective Function, Training Strategy, and Year of publication.

Methods Graph Property Pretext-Task Data Augmentation Objective Function Training Strategy Year
CDNMF Attributed Contrastive/L-C + Generative/AE None InfoNCE + AE URL 2024
Graph Completion Attributed Generative/AE Attribute Masking MAE P&F/JL 2020
Node Attribute Masking Attributed Generative/AE Attribute Masking MAE P&F/JL 2020
Edge Attribute Masking Attributed Generative/AE Attribute Masking MAE P&F 2019
Node Attribute and
Embedding Denoising		 Attributed Generative/AE Attribute Masking MAE JL 2020
Adjacency Matrix
Reconstruction		 Attributed Generative/AE Attribute Masking
Edge Perturbation		 MAE JL 2020
Graph Bert Attributed Generative/AE Attribute Masking
Edge Perturbation		 MAE P&F 2020
Pretrain-Recsys Attributed Generative/AE Edge Perturbation MAE P&F 2021
GPT-GNN Heterogeneous Generative/AR Attribute Masking
Edge Perturbation		 MAE/InfoNCE P&F 2020
GraphCL Attributed Contrastive/G-G Attribute Masking
Edge Perturbation
Random Walk Sampling		 InfoNCE URL 2020
IGSD Attributed Contrastive/G-G Edge Perturbation
Edge Doffisopm		 InfoNCE JL/URL 2020
DACL Attributed Contrastive/G-G Mixup InfoNCE URL 2020
LCC Attributed Contrastive/G-G None InfoNCE JL 2021
CCGL Attributed Contrastive/G-G Information Re-Diffusion InfoNCE P&F 2021
CSSL Attributed Contrastive/G-G NodeInsertion
Edge Perturbation
Uniform Sampling		 InfoNCE P&F/JL/URL 2020
GCC Unattributed Contrastive/C-C Random Walk Sampling InfoNCE P&F/URL 2020
GRACE Attributed Contrastive/L-L Attribute Masking
Edge Perturbation		 InfoNCE URL 2020
GCA Attributed Contrastive/L-L Attention-based InfoNCE URL 2020
GROC Attributed Contrastive/L-L Gradient-based InfoNCE URL 2021
SEPT Attributed Contrastive/L-L Edge Perturbation InfoNCE JL 2021
STDGI Spatial-Temporal Contrastive/L-L Attribute Shuffling JS Estimator URL 2019
GMI Attributed Contrastive/L-L None SP Estimator URL 2020
KS2L Attributed Contrastive/L-L None InfoNCE URL 2020
CG3 Attributed Contrastive/L-L None InfoNCE JL 2020
BGRL Attributed Contrastive/L-L Attribute Masking
Edge Perturbation		 Inner Product URL 2021
SelfGNN Attributed Contrastive/L-L Attribute Masking
Edge Diffusion		 MSE URL 2021
HeCo Heterogeneous Contrastive/L-L None InfoNCE URL 2021
PT-DGNN Dynamic Contrastive/L-L Attribute Masking
Edge Perturbation		 InforNCE P&F 2021
COAD Attributed Contrastive/L-L None Triplet Margin Loss P&F 2020
Contrst-Reg Attributed Contrastive/L-L Attribute Shuffling InfoNCE JL 2021
DGI Attributed Contrastive/L-G Arbitrary JS Estimator URL 2019
HDMI Attributed Contrastive/L-G Attribute Shuffling JS Estimator URL 2021
DMGI Heterogeneous Contrastive/L-G Attribute Shuffling JS Estimator/MAE URL 2020
MVGRL Attributed Contrastive/L-G Attribute Masking
Edge Perturbation
Edge Diffusion
Random Walk Sampling		 DV Estimator
JS Estimator
NT-Xent
InfoNCE		 URL 2020
HDGI Heterogeneous Contrastive/L-G Attribute Shuffling JS Estimator URL 2019
Subg-Con Attributed Contrastive/L-C Importance Sampling Triplet Margin Loss URL 2020
Cotext Prediction Attributed Contrastive/L-C Ego-nets Sampling Cross Entropy P&F 2019
GIC Attributed Contrastive/L-C Arbitrary JS Estimator URL 2020
GraphLoG Attributed Contrastive/L-C Attribute Masking InfoNCE URL 2021
MHCN Heterogeneous Contrastive/L-C Attribute Shuffling InfoNCE JL 2021
EGI Attributed Contrastive/L-C Ego-nets Sampling SP Estimator P&F 2020
MICRO-Graph Attributed Contrastive/C-G Knowledge Sampling InfoNCE URL 2020
InfoGraph Attributed Contrastive/C-G None SP Estimator URL 2019
SUGAR Attributed Contrastive/C-G BFS Sampling JS Estimator JL 2021
BiGI Heterogeneous Contrastive/C-G Edge Perturbation
Ego-nets Sampling		 JS Estimator JL 2021
HTC Attributed Contrastive/C-G Attribute Shuffling SP Estimator
DV Estimator		 URL 2021
Node Property Prediction Attributed Predictive/Node Property None MAE P&F/JL 2020
S2GRL Attributed Predictive/Context-based None Cross Entropy URL 2020
PairwiseDistance Attributed Predictive/Context-based None Cross Entropy P&F/JL 2020
PairwiseAttrSim Attributed Predictive/Context-based None MAE P&F/JL 2020
Distance2Cluster Attributed Predictive/Context-based None MAE P&F/JL 2020
EdgeMask Attributed Predictive/Context-based None Cross Entropy P&F/JL 2020
TopoTER Attributed Predictive/Context-based Edge Perturbation Cross Entropy URL 2021
Centrality Score Ranking Attributed Predictive/Context-based None Cross Entropy P&F 2019
Meta-path prediction Heterogeneous Predictive/Context-based None Cross Entropy JL 2020
SLiCE Heterogeneous Predictive/Context-based None Cross Entropy P&F 2020
Distance2Labeled Attributed Predictive/Context-based None MAE P&F/JL 2020
ContextLabel Attributed Predictive/Context-based None MAE P&F/JL 2020
HCM Attributed Predictive/Context-based Edge Perturbation Bayesian Inference URL 2021
Contextual Molecular
Property Prediction		 Attributed Predictive/Domain-based None Cross Entropy P&F 2020
Graph-level Motif Prediction Attributed Predictive/Domain-based None Cross Entropy P&F 2020
Multi-stage Self-training Attributed Predictive/Self-training None None JL 2018
Node Clustering Attributed Predictive/Self-training None Clustering P&F/JL 2020
Graph Partitioning Attributed Predictive/Self-training None Graph Partitioning P&F/JL 2020
CAGAN Attributed Predictive/Self-training None Clustering URL 2020
M3S Attributed Predictive/Self-training None Clustering JL 2020
Cluster Preserving Attributed Predictive/Self-training None Cross Entropy P&F 2019

A Summary of Implementation Details

About Task Level, Evaluation Metric, and Evaluation Datasets.

Methods Task Level Evaluation Metric Dataset
CDNMF Node Node Clustering (Acc, NMI) Cora, Citeseer, Pubmed
Graph Completion Node Node Classification (Acc) Cora, Citeseer, Pubmed
Node Attribute Masking Node Node Classification (Acc) Cora, Citeseer, Pubmed, Reddit
Edge Attribute Masking Graph Graph Classification (ROC-AUC) MUTAG, PTC, PPI, BBBP, Tox21, ToxCast, ClinTox, MUV, HIV, SIDER, BACE
Node Attribute and
Embedding Denoising		 Node Node Classification (Acc) Cora, Citeseer, Pubmed
Adjacency Matrix
Reconstruction		 Node Node Classification (Acc) Cora, Citeseer, Pubmed
Graph Bert Node Node Classification (Acc)
Node Clustering (NMI)		 Cora, Citeseer, Pubmed
Pretrain-Recsys Node/Link - ML-1M, MOOCs and Last-FM
GPT-GNN Node/Link Node Classification (F1-score)
Link Prediction (ROC-AUC)		 OAG, Amazon, Reddit
GraphCL Graph Graph Classification (Acc, ROC-AUC) NCI1, PROTEINS, D&D, COLLAB, RDT-B, RDT-M5K, GITHUB, MNIST, CIFAR10, MUTAG, IMDB-B, BBBP, Tox21, ToxCast, SIDER, ClinTox, MUV, HIV, BACE, PPI
IGSD Graph Graph Classification (Acc) MUTAG, PTC_MR, NCI1, IMDB-B, QM9, COLLAB, IMDB-M
DACL Graph Graph Classification (Acc) MUTAG, PTC_MR, IMDB-B, IMDB-M, RDT-B, RDT-M5K
LCC Graph Graph Classification (Acc) IMDB-B, IMDB-M, COLLAB, MUTAG, PROTEINS, PTC, NCI1, D&D
CCGL Graph Cascade Graph Prediction (MSLE) Weibo, Twitter, ACM, APS, DBLP
CSSL Graph Graph Classification (Acc) PROTEINS, D&D, NCI1, NCI109, Mutagenicity
GCC Node/Graph Node Classification (Acc)
Graph Classification (Acc)		 US-Airport, H-index, COLLAB, IMDB-B, IMDB-M, RDT-B, RDT-M5K
GRACE Node Node Classification (Acc, Micro-F1) Cora, Citeseer, Pubmed, DBLP, Reddit, PPI
GCA Node Node Classification (Acc) Wiki-CS, Amazon-Computers, Amazon-Photo, Coauthor-CS, Coauthor-Physics
GROC Node Node Classification (Acc) Cora, Citeseer, Pubmed, Amazon-Photo, Wiki-CS
SEPT Node/Link - Last-FM, Douban, Yelp
STDGI Node Node Regression (MAE, RMSE, MAPE) METR-LA
GMI Node/Link Node Classification (Acc, Micro-F1)
Link Prediction (ROC-AUC)		 Cora, Citeseer, PubMed, Reddit, PPI, BlogCatalog, Flickr
KS2L Node/Link Node Classification (Acc)
Link Prediction (ROC-AUC)		 Cora, Citeseer, Pubmed, Amazon-Computers, Amazon-Photo, Coauthor-CS
CG3 Node Node Classification (Acc) Cora, Citeseer, Pubmed, Amazon-Computers, Amazon-Photo, Coauthor-CS
BGRL Node Node Classification (Acc, Micro-F1) Wiki-CS, Amazon-Computers, Amazon-Photo, PPI, Coauthor-CS, Coauthor-Physics, ogbn-arxiv
SelfGNN Node Node Classification (Acc) Cora, Citeseer, Pubmed, Amazon-Computers, Amazon-Photo, Coauthor-CS, Coauthor-Physics
HeCo Node Node Classification
(ROC-AUC, Micro-F1, Macro-F1)
Node Clustering (NMI, ARI)		 ACM, DBLP, Freebase, AMiner
PT-DGNN Link Link Prediction (ROC-AUC) HepPh, Math Overflow, Super User
COAD Node/Link Node Clustering
(Precision, Recall, F1-score)
Link Prediction (HitRatio@K, MRR)		 AMiner, News, LinkedIn
Contrast-Reg Node/Link Node Classification (Acc)
Node Clustering
(NMI, Acc, Macro-F1)
Link Prediction (ROC-AUC)		 Cora, Citeseer, Pubmed, Reddit, ogbn-arxiv, Wikipedia, ogbn-products, Amazo-Computers, Amazo-Photo
DGI Node Node Classification (Acc, Micro-F1) Cora, Citeseer, Pubmed, Reddit, PPI
HDMI Node Node Classification
(Micro-F1, Macro-F1)
Node Clustering (NMI)		 ACM, IMDB, DBLP, Amazon
DMGI Node Node Clustering (NMI)
Node Classification (Acc)		 ACM, IMDB, DBLP, Amazon
MVGRL Node/Graph Node Classification (Acc)
Node Clustering (NMI, ARI)
Graph Classification (Acc)		 Cora, Citeseer, Pubmed, MUTAG, PTC_MR, IMDB-B, IMDB-M, RDT-B
HDGI Node Node Classification
(Micro-F1, Macro-F1)
Node Clustering (NMI, ARI)		 ACM, DBLP, IMDB
Subg-Con Node Node Classification (Acc, Micro-F1) Cora, Citeseer, Pubmed, PPI, Flickr, Reddit
Cotext Prediction Graph Graph Classification (ROC-AUC) MUTAG, PTC, PPI, BBBP, Tox21, ToxCast, ClinTox, MUV, HIV, SIDER, BACE
GIC Node/Link Node Classification (Acc)
Node Clustering (Acc, NMI, ARI)
Link Prediction (ROC-AUC, ROC-AP)		 Cora, Citeseer, Pubmed, Amazon-Computers, Amazon-Photo, Coauthor-CS, Coauthor-Physics
GraphLoG Graph Graph Classification (ROC-AUC) BBBP, Tox21, ToxCast, ClinTox, MUV, HIV, SIDER, BACE
MHCN Node/Link - Last-FM, Douban, Yelp
EGI Node/Link Node Classification (Acc)
Link Prediction (ROC-AUC, MRR)		 YAGO, Airport
MICRO-Graph Graph Graph Classification (ROC-AUC) BBBP, Tox21, ToxCast, ClinTox, HIV, SIDER, BACE
InfoGraph Graph Graph Classification (Acc) MUTAG, PTC_MR, RDT-B, RDT-M5K, IMDB-B, QM9, IMDB-M
SUGAR Graph Graph Classification (Acc) MUTAG, PTC, PROTEINS, D&D, NCI1, NCI109
BiGI Link Link Prediction (AUC-ROC, AUC-PR) DBLP, ML-100K, ML-1M, Wikipedia
HTC Graph Graph Classification (Acc) MUTAG, PTC_MR, IMDB-B, IMDB-M, RDT-B, QM9, RDT-M5K
Node Property Prediction Node Node Classification (Acc) Cora, Citeseer, Pubmed, Reddit
S2GRL Node/Link Node Classification (Acc, Micro-F1)
Node Clustering (NMI)
Link Prediction (ROC-AUC)		 Cora, Citeseer, Pubmed, PPI, Flickr, BlogCatalog, Reddit
PairwiseDistance Node Node Classification (Acc) Cora, Citeseer, Pubmed, Reddit
PairwiseAttrSim Node Node Classification (Acc) Cora, Citeseer, Pubmed, Reddit
Distance2Cluster Node Node Classification (Acc) Cora, Citeseer, Pubmed, Reddit
EdgeMask Node Node Classification (Acc) Cora, Citeseer, Pubmed, Reddit
TopoTER Node/Graph Node Classification (Acc)
Graph Classification (Acc)		 Cora, Citeseer, Pubmed, MUTAG, PTC-MR, RDT-B, RDT-M5K, IMDB-B, IMDB-M
Centrality Score Ranking Node/Link/Graph Node Classification (Micro-F1)
Link Prediction (Micro-F1)
Graph Classification (Micro-F1)		 Cora, Pubmed, ML-100K, ML-1M, IMDB-M, IMDB-B
Meta-path prediction Node/Link Node Classification (F1-score)
Link Prediction (ROC-AUC)		 ACM, IMDB, Last-FM, Book-Crossing
SLiCE Link Link Prediction (ROC-AUC, Micro-F1) Amazon, DBLP, Freebase, Twitter, Healthcare
Distance2Labeled Node Node Classification (Acc) Cora, Citeseer, Pubmed, Reddit
ContextLabel Node Node Classification (Acc) Cora, Citeseer, Pubmed, Reddit
HCM Node Node Classification (ROC-AUC) ACM, Amazon, Enron, BlogCatalog, Flickr
Contextual Molecular
Property Prediction		 Graph Graph Classification (Acc)
Graph Regression (MAE)		 BBBP, SIDER, ClinTox, BACE, Tox21, ToxCast, ESOL, FreeSolv, Lipo, QM7, QM8
Graph-level Motif Prediction Graph Graph Classification (Acc)
Graph Regression (MAE)		 BBBP, SIDER, ClinTox, BACE, Tox21, ToxCast, ESOL, FreeSolv, Lipo, QM7, QM8
Multi-stage Self-training Node Node Classification (Acc) Cora, Citeseer, Pubmed
Node Clustering Node Node Classification (Acc) Cora, Citeseer, Pubmed
Graph Partitioning Node Node Classification (Acc) Cora, Citeseer, Pubmed
CAGAN Node Node Classfication
(Micro-F1, Macro-F1)
Node Clustering
(Micro-F1, Macro-F1, NMI)		 Cora, Citeseer, Pubmed
M3S Node Node Classification (Acc) Cora, Citeseer, Pubmed
Cluster Preserving Node/Link/Graph Node Classification (Micro-F1)
Link Prediction (Micro-F1)
Graph Classification (Micro-F1)		 Cora, Pubmed, ML-100K, ML-1M, IMDB-M, IMDB-B

A Summary of Common Graph Datasets

About category, graph number, node number per graph, edge number per graph, dimensionality of node attributes, class number, and citation papers.

Dataset Category #Graph #Node (Avg.) #Edge (Avg.) #Feature #Class
Cora Citation Network 1 2708 5429 1433 7
Citeseer Citation Network 1 3327 4732 3703 6
Pubmed Citation Network 1 19717 44338 500 3
Wiki-CS Citation Network 1 11701 216123 300 10
Coauthor-CS Citation Network 1 18333 81894 6805 15
Coauthor-Physics Citation Network 1 34493 247962 8415 5
DBLP (v12) Citation Network 1 4894081 45564149 - -
ogbn-arxiv Citation Network 1 169343 1166243 128 40
Reddit Social Network 1 232965 11606919 602 41
BlogCatalog Social Network 1 5196 171743 8189 6
Flickr Social Network 1 7575 239738 12047 9
COLLAB Social Networks 5000 74.49 2457.78 - 2
RDT-B Social Networks 2000 429.63 497.75 - 2
RDT-M5K Social Networks 4999 508.52 594.87 - 5
IMDB-B Social Networks 1000 19.77 96.53 - 2
IMDB-M Social Networks 1500 13.00 65.94 - 3
ML-100K Social Networks 1 2625 100000 - 5
ML-1M Social Networks 1 9940 1000209 - 5
PPI Protein Networks 24 56944 818716 50 121
D&D Protein Networks 1178 284.32 715.65 82 2
PROTEINS Protein Networks 1113 39.06 72.81 4 2
NCI1 Molecule Graphs 4110 29.87 32.30 37 2
MUTAG Molecule Graphs 188 17.93 19.79 7 2
QM9 (QM7, QM8) Molecule Graphs 133885 - - - -
BBBP Molecule Graphs 2039 24.05 25.94 - 2
Tox21 Molecule Graphs 7831 18.51 25.94 - 12
ToxCast Molecule Graphs 8575 18.78 19.26 - 167
ClinTox Molecule Graphs 1478 26.13 27.86 - 2
MUV Molecule Graphs 93087 24.23 26.28 - 17
HIV Molecule Graphs 41127 25.53 27.48 - 2
SIDER Molecule Graphs 1427 33.64 35.36 - 27
BACE Molecule Graphs 1513 34.12 36.89 - 2
PTC Molecule Graphs 344 14.29 14.69 19 2
NCI109 Molecule Graphs 4127 29.68 32.13 - 2
Mutagenicity Molecule Graphs 4337 30.32 30.77 - 2
MNIST Others (Image) - 70000 - 784 10
CIFAR10 Others (Image) - 60000 - 1024 10
METR-LA Others (Traffic) 1 207 1515 2 -
Amazon-Computers Others (Purchase) 1 13752 245861 767 10
Amazon-Photo Others (Purchase) 1 7650 119081 745 8
ogbn-products Others (Purchase) 1 2449029 61859140 100 47

A Summary of Open-source Codes

Methods Github
CDNMF https://github.com/6lyc/CDNMF
Graph Completion https://github.com/Shen-Lab/SS-GCNs
Node Attribute Masking https://github.com/ChandlerBang/SelfTask-GNN
Edge Attribute Masking http://snap.stanford.edu/gnn-pretrain
Attribute and Embedding Denoising N.A.
Adjacency Matrix Reconstruction N.A.
Graph Bert https://github.com/anonymous-sourcecode/Graph-Bert
Pretrain-Recsys https://github.com/jerryhao66/Pretrain-Recsys
SLAPS https://github.com/BorealisAI/SLAPS-GNN
G-BERT https://github.com/jshang123/G-Bert
GPT-GNN https://github.com/acbull/GPT-GNN
GraphCL https://github.com/Shen-Lab/GraphCL
IGSD N.A.
DACL N.A.
LCC https://github.com/YuxiangRen
CCGL https://github.com/Xovee/ccgl
CSSL N.A.
GCC https://github.com/THUDM/GCC
GRACE https://github.com/CRIPAC-DIG/GRACE
GCA https://github.com/CRIPAC-DIG/GCA
GROC N.A.
SEPT https://github.com/Coder-Yu/QRec
STDGI N.A.
GMI https://github.com/zpeng27/GMI
KS2L N.A.
CG3 N.A.
BGRL N.A.
SelfGNN https://github.com/zekarias-tilahun/SelfGNN
HeCo https://github.com/liun-online/HeCo
PT-DGNN https://github.com/Mobzhang/PT-DGNN
COAD https://github.com/allanchen95/Expert-Linking
Contrast-Reg N.A.
C-SWM https://github.com/tkipf/c-swm
DGI https://github.com/PetarV-/DGI
HDMI N.A.
DMGI https://github.com/pcy1302/DMGI
MVGRL https://github.com/kavehhassani/mvgrl
HDGI https://github.com/YuxiangRen/Heterogeneous-Deep-Graph-Infomax
Subg-Con https://github.com/yzjiao/Subg-Con
Cotext Prediction http://snap.stanford.edu/gnn-pretrain
GIC https://github.com/cmavro/Graph-InfoClust-GIC
GraphLoG https://openreview.net/forum?id=DAaaaqPv9-q
MHCN https://github.com/Coder-Yu/RecQ
EGI https://openreview.net/forum?id=J_pvI6ap5Mn
MICRO-Graph https://drive.google.com/file/d/1b751rpnV-SDmUJvKZZI-AvpfEa9eHxo9/
InfoGraph https://github.com/fanyun-sun/InfoGraph
SUGAR https://github.com/RingBDStack/SUGAR
BiGI https://github.com/clhchtcjj/BiNE
HTC N.A.
DITNET https://github.com/FangpingWan/NeoDTI
Node Property Prediction https://github.com/ChandlerBang/SelfTask-GNN
S2GRL N.A.
PairwiseDistance https://github.com/ChandlerBang/SelfTask-GNN
PairwiseAttrSim https://github.com/ChandlerBang/SelfTask-GNN
Distance2Cluster https://github.com/ChandlerBang/SelfTask-GNN
EdgeMask https://github.com/ChandlerBang/SelfTask-GNN
TopoTER N.A.
Centrality Score Ranking N.A.
Meta-path prediction https://github.com/mlvlab/SELAR
SLiCE https://github.com/pnnl/SLICE
Distance2Labeled https://github.com/ChandlerBang/SelfTask-GNN
ContextLabel https://github.com/ChandlerBang/SelfTask-GNN
HCM N.A.
Contextual Molecular Property Prediction https://github.com/tencent-ailab/grover
Graph-level Motif Prediction https://github.com/tencent-ailab/grover
DrRepair https://github.com/michiyasunaga/DrRepair
Multi-stage Self-training https://github.com/Davidham3/deeper_insights_into_GCNs
Node Clustering https://github.com/Shen-Lab/SS-GCNs
Graph Partitioning https://github.com/Shen-Lab/SS-GCNs
CAGAN N.A.
M3S https://github.com/datake/M3S
Cluster Preserving N.A.
SEF https://github.com/nealgravindra/self-supervsed_edge_feats

Contribute

If you would like to help contribute this list, please feel free to contact me or add pull request with the following Markdown format:

- Paper Name. 
  - Author List. *Conference Year*. [[pdf]](link) [[code]](link)

This is a Github Summary of our Survey. If you find this file useful in your research, please consider citing:

@article{wu2021self,
  title={Self-supervised Learning on Graphs: Contrastive, Generative, or Predictive},
  author={Wu, Lirong and Lin, Haitao and Tan, Cheng and Gao, Zhangyang and Li, Stan Z},
  journal={IEEE Transactions on Knowledge and Data Engineering},
  year={2021},
  publisher={IEEE}
}

Feedback

If you have any issue about this work, please feel free to contact me by email: