Initial Release of Captum for Model Interpretability, Attribution and Debugging

We just released our first version of the PyTorch Captum library for model interpretability!

Highlights

This first release, v0.1.0, supports a number of gradient-based attribution algorithms as well as Captum Insights, a visualization tool for model debugging and understanding.

Attribution Algorithms

The following general purpose gradient-based attribution algorithms are provided. These can be applied to any type of PyTorch model and input features, including image, text, and multimodal.

1. Attribution of output of the model with respect to the input features

   1. Saliency
   2. InputXGradient
   3. IntegratedGradient
   4. DeepLift
   5. DeepLiftShap
   6. GradientShap

2. Attribution of output of the model with respect to the layers of the model

   1. LayerActivation
   2. LayerGradientXActivation
   3. LayerConductance
   4. InternalInfluence

3. Attribution of neurons with respect to the input features

   1. NeuronGradient
   2. NeuronIntegratedGradients
   3. NeuronConductance

4. Attribution Algorithm + noisy sampling

   1. NoiseTunnel
      NoiseTunnel helps to reduce the noise in the attributions that are assigned by attribution algorithms by using different noise tunnel techniques such as smoothgrad, smoothgrad_sq and vargrad.

Batch and Data Parallel Optimizations

Since some of the algorithms, like integrated gradients, expand input tensors internally, we want to make sure we can scale those tensors and our forward/backward computations efficiently. For that reason, we developed a feature that chunks tensors internally into internal_batch_size pieces, an argument which can be passed as input to attribute methods, which will make the library run forward and backward passes for each tensor batch separately and ultimately combine those after computing gradients.

The algorithms that support batched optimization are:

1. IntegratedGradients
2. LayerConductance
3. InternalInfluence
4. NeuronConductance

PyTorch data parallel models are also supported across all Captum algorithms, allowing users to take advantage of multiple GPUs when applying interpretability algorithms.

More details on these algorithms can be found on our website at captum.ai/docs/algorithms

Captum Insights

Captum Insights provides these algorithms in an interactive Jupyter notebook-based tool for model debugging and understanding. It can be used embedded within a notebook or run as a standalone application.

Features:

1. Visualize attribution across sampled data for classification models
2. Multimodal support with text, image and general features into a single model
3. Filtering and debugging specific sets of classes and misclassified examples
4. Jupyter notebook support for easy model and dataset modification

Insights is built with standard web technologies including JavaScript, CSS, React, Yarn and Flask.