02 Network manipulation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Network Manipulation\n",
    "\n",
    "In this part, we are going to learn to manipulate a network, including forward\n",
    "and backward passes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\"\"\"Initialization (see \"00 Basic solver usage\").\"\"\"\n",
    "import os\n",
    "import numpy as np\n",
    "\n",
    "# Silence caffe network loading output. Must be set before importing caffe\n",
    "os.environ[\"GLOG_minloglevel\"] = '2'\n",
    "import caffe\n",
    "CAFFE_ROOT=\"/caffe\"\n",
    "os.chdir(CAFFE_ROOT) # change the current directory to the caffe root, to help\n",
    "                     # with the relative paths\n",
    "USE_GPU = True\n",
    "if USE_GPU:\n",
    "    caffe.set_device(0)\n",
    "    caffe.set_mode_gpu()\n",
    "else:\n",
    "    caffe.set_mode_cpu()\n",
    "# For reproducible results\n",
    "caffe.set_random_seed(0) # recent modification, remove if it doesn't work\n",
    "np.random.seed(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're going to use the basic MNIST example, with the adapted _LeNet_ network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "net = caffe.Net(\"examples/mnist/lenet_train_test.prototxt\", caffe.TRAIN)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inspecting the network\n",
    "\n",
    "You can inspect the network, to see the layer names (`net._layer_names`), or the\n",
    "list of layers (`net.layers`) with their types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Network layers:\n",
      "mnist  : Data             (0 blobs)\n",
      "conv1  : Convolution      (2 blobs)\n",
      "pool1  : Pooling          (0 blobs)\n",
      "conv2  : Convolution      (2 blobs)\n",
      "pool2  : Pooling          (0 blobs)\n",
      "ip1    : InnerProduct     (2 blobs)\n",
      "relu1  : ReLU             (0 blobs)\n",
      "ip2    : InnerProduct     (2 blobs)\n",
      "loss   : SoftmaxWithLoss  (0 blobs)\n"
     ]
    }
   ],
   "source": [
    "print(\"Network layers:\")\n",
    "for name, layer in zip(net._layer_names, net.layers):\n",
    "    print(\"{:<7}: {:17s}({} blobs)\".format(name, layer.type, len(layer.blobs)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To access the weights of a layer, simply access the blobs of that layer. Usually, there are either no blobs (inputs, ReLU, Pooling, loss) or 2 blobs, the weights and the bias. You can modify them as you would modify a numpy array.\n",
    "\n",
    "Another way to access those parameters are through `net.params`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(net.params[\"ip1\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To inspect the data flow, and the gradients over these flows, you can inspect the network blobs. It is an `OrderedDict` (bottom to top) of blob names to blobs. These blobs carry the data between the layers (see http://caffe.berkeleyvision.org/tutorial/net_layer_blob.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Blobs:\n",
      "data :  (64, 1, 28, 28)\n",
      "label:  (64,)\n",
      "conv1:  (64, 20, 24, 24)\n",
      "pool1:  (64, 20, 12, 12)\n",
      "conv2:  (64, 50, 8, 8)\n",
      "pool2:  (64, 50, 4, 4)\n",
      "ip1  :  (64, 500)\n",
      "ip2  :  (64, 10)\n",
      "loss :  ()\n"
     ]
    }
   ],
   "source": [
    "print(\"Blobs:\")\n",
    "for name, blob in net.blobs.iteritems():\n",
    "    print(\"{:<5}:  {}\".format(name, blob.data.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The blobs provide access to:\n",
    "- `num`: the number of elements (usually `batch_size`)\n",
    "- `channels`, `height`, `width`: the dimension of a sample\n",
    "- `shape`: a tuple with `(num, channels, height, width)`\n",
    "- `count`: `num * channels * height * width`\n",
    "- `data`: the data stored in the blob (see forward propagation)\n",
    "- `diff`: the computed gradient for the blob (see backward propagation)\n",
    "\n",
    "Be aware that when getting the `data` or `diff` of a blob, it is a reference, or pointer, to the actual memory, so anything changing it will change all current references. To get a snapshot of the value that will not change, copy it with `blob.data.copy()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Blob attributes:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['channels',\n",
       " 'count',\n",
       " 'data',\n",
       " 'diff',\n",
       " 'height',\n",
       " 'num',\n",
       " 'reshape',\n",
       " 'shape',\n",
       " 'width']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"Blob attributes:\")\n",
    "[e for e in dir(net.blobs[\"label\"]) if not e.startswith(\"__\")]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To examine the links between the layers, you can access the lists of bottom/top names of the layers. `net.bottom_names` and `net.top_names` contain pseudo-dictionnaries that return the list of blob names that are the bottom/tops of the layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['data', 'label']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net.top_names[\"mnist\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['ip1']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net.top_names[\"ip1\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['ip2', 'label']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net.bottom_names[\"loss\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The inputs of the network (layers of type `Input`, not LMDB or HDF5 or other input types), and the outputs of the network (layers whose `top`s are no one's `bottom`) can be accessed with `net.inputs` and `net.outputs`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['data', 'label']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net.inputs # No inputs, since our input layer is of type \"Data\", not \"Input\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['loss']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net.outputs # In testing mode, we would also have 'accuracy'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Forward propagation\n",
    "\n",
    "Now, let's run an example through the network. \n",
    "\n",
    "As we haven't covered how to\n",
    "load the data yet, we're going to generate a random batch with values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "batch = np.random.randn(*net.blobs[\"data\"].shape) * 50 # normal distribution(0, 50), in the shape of the input batch\n",
    "labels = np.random.randint(0, 10, net.blobs[\"label\"].shape) # random labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To provide an input for the network, we are going to fill the `data` field of\n",
    "the input layer with our batch.\n",
    "\n",
    "We cannot assign directly the field `data`, since it is a C++ field. However,\n",
    "we can set its contents, like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "net.blobs[\"data\"].data[...] = batch\n",
    "net.blobs[\"label\"].data[...] = labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Running the forward pass is then just a matter of calling `net.forward()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'loss': array(2.3706960678100586, dtype=float32)}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net.forward()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is also possible to do a partial forward pass by specifying the start and end layers of the forward pass:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "res = net.forward(start=\"mnist\", end=\"conv1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the network had `Input` layers (layers of type `Input`), then we could have directly called `net.forward` with the data, as such:\n",
    "\n",
    "```python\n",
    "net.forward(data=batch, label=labels)\n",
    "```\n",
    "\n",
    "`data` and `label` are the blob names, and `batch` and `labels` are the values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output of the network can be consulted at any layer (or blob) by inspecting\n",
    "the `data` field."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The predictions for each class are given by the output of the last fully\n",
    "connected layer, `ip2`. To get a single prediction, take the maximum of this\n",
    "array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.48029846,  0.61164796, -0.92312622, -0.47545347, -0.46422586,\n",
       "       -0.03909576,  0.40830323, -0.50112289, -0.07321799,  0.5741204 ], dtype=float32)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net.blobs[\"ip2\"].data[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Output of the loss layer. This is what is minimized during training. If all\n",
    "goes well, this should decrease during training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(2.3706960678100586, dtype=float32)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net.blobs[\"loss\"].data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Backward propagation\n",
    "\n",
    "The backward propagation computes the gradient for all the weights of the layers, and all the data blobs of the network, and stores them in the corresponding `diff`. It does not update the weights, but only computes the gradients.\n",
    "\n",
    "The initial values of the diffs represent the function we are differentiating. The basic setup is to set all the diffs to 0, except the loss layer to 1. This will ensure that the gradients are computed with respect to the loss function. These coefficients are stored in `net.blob_loss_weights`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('data', 0.0),\n",
       "             ('label', 0.0),\n",
       "             ('conv1', 0.0),\n",
       "             ('pool1', 0.0),\n",
       "             ('conv2', 0.0),\n",
       "             ('pool2', 0.0),\n",
       "             ('ip1', 0.0),\n",
       "             ('ip2', 0.0),\n",
       "             ('loss', 1.0)])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net.blob_loss_weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we can just run `net.backward()` to compute the diffs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ -1.01645060e-01,  -6.20732643e-03,  -6.90511540e-02, ...,\n",
       "         -4.16178778e-02,  -5.50730200e-03,  -1.02949061e-03],\n",
       "       [  5.90243051e-03,   6.79359585e-03,  -5.63192554e-03, ...,\n",
       "          1.76133737e-02,  -1.25561077e-02,  -7.78268659e-05],\n",
       "       [ -5.64169660e-02,   1.90727238e-04,  -3.76255438e-02, ...,\n",
       "         -2.53438223e-02,  -1.07933357e-02,   7.93526197e-05],\n",
       "       ..., \n",
       "       [ -5.50880432e-02,  -9.05541796e-03,  -1.71823744e-02, ...,\n",
       "         -2.88205203e-02,   3.69150680e-03,   1.57054863e-04],\n",
       "       [  1.49490433e-02,  -6.27137860e-03,   1.67495478e-02, ...,\n",
       "          3.13645788e-03,  -2.81939493e-03,   1.18995311e-04],\n",
       "       [  1.03935905e-01,   2.79635563e-03,   3.92960571e-02, ...,\n",
       "          3.40057276e-02,   1.03832046e-02,   1.61820513e-04]], dtype=float32)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net.backward()\n",
    "net.layers[list(net._layer_names).index(\"ip2\")].blobs[0].diff # Gradient for the parameters of the ip2 layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we had two loss layers, we could weight one more than the other by settings the diffs. Or to optimize for a specific output, we could set the diffs manually. For instance, to compute the gradients to optimize the output of class 1, we could do:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 69.95400238,   3.21474099,  39.70861435, ...,  23.07016373,\n",
       "          4.77206326,   0.09005913],\n",
       "       [  0.        ,   0.        ,   0.        , ...,   0.        ,\n",
       "          0.        ,   0.        ],\n",
       "       [  0.        ,   0.        ,   0.        , ...,   0.        ,\n",
       "          0.        ,   0.        ],\n",
       "       ..., \n",
       "       [  0.        ,   0.        ,   0.        , ...,   0.        ,\n",
       "          0.        ,   0.        ],\n",
       "       [  0.        ,   0.        ,   0.        , ...,   0.        ,\n",
       "          0.        ,   0.        ],\n",
       "       [  0.        ,   0.        ,   0.        , ...,   0.        ,\n",
       "          0.        ,   0.        ]], dtype=float32)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We need to clear the previously computed diffs from the layers, otherwise they are just added\n",
    "for l in net.layers:\n",
    "    for b in l.blobs:\n",
    "        b.diff[...] = 0\n",
    "        \n",
    "d = net.blobs[\"ip2\"].diff # Top of the ip2 layer\n",
    "d[...] = 0 # Clear the diff\n",
    "d[:, 0] = 1 # Optimize for each element of the batch, for class 1 (indexes are 0-based)\n",
    "net.backward(start=\"ip2\") # Start the backpropagation at the ip2 layer, working down\n",
    "net.layers[list(net._layer_names).index(\"ip2\")].blobs[0].diff"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Updating the weights\n",
    "\n",
    "To update the weights, we have to apply the diffs to the layer weights, affected by a learning rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "lr = 0.01\n",
    "for l in net.layers:\n",
    "    for b in l.blobs:\n",
    "        b.data[...] -= lr * b.diff"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have all the elements, we will move on to writing our own custom solver."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}