Session_7_Intro_to_Deep_Learning.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyMZ8w4ZtzG5Jco8ZPDN2HSR",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/stevengiacalone/Python-workshop/blob/main/Session_7_Intro_to_Deep_Learning.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# An Introduction to Deep Learning"
      ],
      "metadata": {
        "id": "B5nwENSgJA11"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "In today's session, we'll dive into the world of machine learning. If this is your first exposure to the concept, you've probably heard many words that kind of sound like the same thing: artificial intelligence, machine learning, neural networks, deep learning, etc. What's the difference? These concepts are arranged in the following hierarchy:\n",
        "\n",
        "<div>\n",
        "<img src=\"http://apmonitor.com/do/uploads/Main/ai_overview.png\" width=\"500\"/>\n",
        "</div>"
      ],
      "metadata": {
        "id": "jd1nlEUPJFJi"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's go over each of these concepts quickly.\n",
        "\n",
        "## Artificial Intelligence\n",
        "\n",
        "Artificial Intelligence is the field of developing computers and robots that are capable of behaving in ways that both mimic and go beyond human capabilities. AI-enabled programs can analyze and contextualize data to provide information or automatically trigger actions without human interference. [Source](https://ai.engineering.columbia.edu/ai-vs-machine-learning/)"
      ],
      "metadata": {
        "id": "iROp7g_0Kj-O"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Machine Learning\n",
        "\n",
        "Machine learning is a pathway to artificial intelligence. This subcategory of AI uses algorithms to automatically learn insights and recognize patterns from data, applying that learning to make increasingly better decisions. [Source](https://ai.engineering.columbia.edu/ai-vs-machine-learning/)\n",
        "\n",
        "Broadly speaking, machine learning algorithms can be divided into two categories: **supervised learning** and **unsupervised learning**. In supervised learning, you train your model on *labeled* data (e.g., images that you know the correct classifications of). Supervised machine learning algorithms are often trained for the purpose of **regression** (i.e., predicting a continuous value) or **classification** (i.e., predicting a categorical value). In unsupervised learning, you train your model on *unlabeled* data, usually to identify unknown correlations and trends.\n",
        "\n",
        "Here is a useful visualization of the difference between the two (from [here](https://www.labellerr.com/blog/supervised-vs-unsupervised-learning-whats-the-difference/)):\n",
        "\n",
        "<div>\n",
        "<img src=\"https://www.labellerr.com/blog/content/images/size/w2000/2023/02/bannerSupervised-vs.-Unsupervised-Learning-1.webp\" width=\"700\"/>\n",
        "</div>\n",
        "\n",
        "There are many types of machine learning algorithms that we will not have time to talk about here, but I encourage you to check them out if the topic interests you!\n",
        "\n",
        "- Linear regression\n",
        "- Naive Bayes\n",
        "- K Nearest Neighbors\n",
        "- Support Vector Machine\n",
        "- Decision trees and random forests\n",
        "- many more!"
      ],
      "metadata": {
        "id": "ywgO_0j4R3Ki"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Neural Networks\n",
        "\n",
        "A neural network is a machine learning agorithm inspired by the human brain. In the brain, groups of interconnected units called neurons send signals to one another. Many of them combined can perform complex tasks. In the context of machine learning, these tasks can include regression and classification.\n",
        "\n",
        "The simplest neural network, known as a **single-layer perceptron**, looks like this:\n",
        "\n",
        "<div>\n",
        "<img src=\"http://apmonitor.com/do/uploads/Main/neural_network.png\" width=\"500\"/>\n",
        "</div>\n",
        "\n",
        "In this example, the perceptron is *fully connected*, meaing every input is connected to every node. Inputs are combined with weights and summed in each node. The summed values are then sent to the output where they are put through an *activation function*, which determines their final value. Here's a visualization of how it works (sorry about the hard-to-read text):\n",
        "\n",
        "<div>\n",
        "<img src=\"https://static.javatpoint.com/tutorial/tensorflow/images/single-layer-perceptron-in-tensorflow2.png\" width=\"800\"/>\n",
        "</div>\n",
        "\n",
        "There are many choices for the [activation function](https://en.wikipedia.org/wiki/Activation_function), but the exact choice will usually depend on the the specifics of you're problem. Some popular activation functions include:\n",
        "- linear\n",
        "- [rectified linear unit](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)) (a.k.a. ReLU)\n",
        "- [logistic](https://en.wikipedia.org/wiki/Logistic_function)\n",
        "- [sigmoid](https://en.wikipedia.org/wiki/Sigmoid_function)\n",
        "\n",
        "When you *train* a neural network, you use an algorithm like [backpropogation](https://en.wikipedia.org/wiki/Backpropagation) to *optimize the values of the weights*."
      ],
      "metadata": {
        "id": "NtjeBkeER4vI"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Deep Learning\n",
        "\n",
        "A deep learning algorithm refers to any neural network with multiple layers of neurons. The simplest deep learning network is the **multi-layer perceptron**, shown below. The layers of neurons are known as *hidden layers*.\n",
        "\n",
        "<div>\n",
        "<img src=\"http://apmonitor.com/do/uploads/Main/deep_neural_network.png\" width=\"700\"/>\n",
        "</div>\n",
        "\n",
        "Each connection (line) in the diagram above has a corresponding weight that is optimized when the model is trained.\n",
        "\n",
        "There are tons of types of deep networks. Some examples are shown in the graphic below (Source). The type of network you use typically depends on the problem that needs solving.\n",
        "\n",
        "<div>\n",
        "<img src=\"https://miro.medium.com/v2/resize:fit:2000/format:webp/1*cuTSPlTq0a_327iTPJyD-Q.png\" width=\"800\"/>\n",
        "</div>\n",
        "\n",
        "In this session, we'll be focusing on one specific deep learning architecture: the **Convolutional Neural Network** ([CNN](https://en.wikipedia.org/wiki/Convolutional_neural_network)). CNNs were first popularized for their ability to perform highly accurate image classification. In the exercise below, we'll be designing and training our own CNN using `tensorflow`, a popular Python package for designing neural networks. But first, let's go over the architecture of a CNN so that we know what's going on in the code."
      ],
      "metadata": {
        "id": "YtQs_bz2fC7X"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Convolutional Neural Networks\n",
        "\n",
        "A CNN is a neural network that utilizes a combination of convolutional layers, pool layers, and dense (i.e., fully connected) layers. We'll go over each of these terms in a moment. Here is a graphic of a typical CNN ([Source](https://www.mdpi.com/1099-4300/19/6/242)):\n",
        "\n",
        "<div>\n",
        "<img src=\"https://www.mdpi.com/entropy/entropy-19-00242/article_deploy/html/images/entropy-19-00242-g001.png\" width=\"800\"/>\n",
        "</div>\n",
        "\n",
        "CNNs can applied on data with any number of dimensions, but in the exercise below we'll use it on 2D data (i.e., images).\n",
        "\n",
        "#### Convolutional Layers\n",
        "\n",
        "A convolutional layer is a layer in which an NxN convolutional filter passes along the input matrix. This is best explained via visualization. Imagine you have a convolutional filter (in this case, a 2x2 matrix) that looks like this ([Source](https://developers.google.com/machine-learning/glossary/#convolutional-layer)):\n",
        "\n",
        "<div>\n",
        "<img src=\"https://developers.google.com/static/machine-learning/glossary/images/ConvolutionalLayerFilter.svg\" width=\"180\"/>\n",
        "</div>\n",
        "\n",
        "And an input matrix that looks like this:\n",
        "\n",
        "<div>\n",
        "<img src=\"https://developers.google.com/static/machine-learning/glossary/images/ConvolutionalLayerInputMatrix.svg\" width=\"500\"/>\n",
        "</div>\n",
        "\n",
        "The filter is passed scanned along the input matrix like so to reduce the size of the input layer. At each location, we are multiplying each pixel of the input matrix by the corresponding filter pixel value, then summing everything together. So for the first position (the top left), we do this:\n",
        "\n",
        "\n",
        "<div>\n",
        "<img src=\"https://developers.google.com/static/machine-learning/glossary/images/ConvolutionalLayerOperation.svg\" width=\"700\"/>\n",
        "</div>\n",
        "\n",
        "This is repeated until you end up with a new, smaller matrix. For an input matrix of size NxN and a convolution filter of size MxM, the output matrix size will be LxL, where L = N - M - 1. Click [here](https://developers.google.com/static/machine-learning/glossary/images/AnimatedConvolution.gif) for a good animation of this process (which does not display well in Colab).\n",
        "\n",
        "#### Pooling Layers\n",
        "\n",
        "A pooling layer reduces the size of the previous layer by \"pooling\" in windows of NxN. Most often, the maximum or average of values in the pooling window are take. Here's a visualization ([Source](https://developers.google.com/machine-learning/glossary/#pooling)):\n",
        "\n",
        "<div>\n",
        "<img src=\"https://developers.google.com/static/machine-learning/glossary/images/PoolingConvolution.svg\" width=\"800\"/>\n",
        "</div>\n",
        "\n",
        "#### Dense Layers\n",
        "\n",
        "A \"dense layer\" is just another term for a group of fully connected hidden layers. In CNNs, these are typically applied right at the end of the algorithm, before the output layer.\n",
        "\n",
        "#### Hyperparameters\n",
        "\n",
        "In any neural network, there will be a number of knobs to turn to change the architecture (and therefore performance) of the algorithm. These knobs are called *hyperparameters*. They range from everything to the number of convolution and pooling layers, to the size of the convolution filter, to the number of neurons in the dense layer, etc etc etc (you get the gist). There is no easy way to determine the optimal hyperparameters for your model, you just need to try a bunch to see what works best."
      ],
      "metadata": {
        "id": "UE9cTUjeKgjF"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Exercise\n",
        "\n",
        "In this exercise, we will be training a CNN to classify images from the MNIST (Modified National Institute of Standards and Technology database), which contains many hand-written images of the digits 0-9. The dataset is often used to test the performance of image classification algorithms.\n",
        "\n",
        "\n",
        "<div>\n",
        "<img src=\"https://camo.githubusercontent.com/960ef680c26a95405b35e9e417955032cb0b0815b746cefd8ff95c73a49ae4ee/687474703a2f2f692e7974696d672e636f6d2f76692f3051493378675875422d512f687164656661756c742e6a7067\" width=\"500\"/>\n",
        "</div>\n",
        "\n",
        "We'll start by importing our Python packages and the data. The data is split into four categories:\n",
        "\n",
        "- train_images: the images that we use to train the algorithm\n",
        "- train_labels: the correct label corresponding to each image (i.e., the actual digit in the image)\n",
        "- test_images: the images that we use to test the trained model\n",
        "- test_labels: the correct label corresponding to each test images"
      ],
      "metadata": {
        "id": "g5b0sbwZSZoW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "from tensorflow.keras.layers import Conv2D, MaxPooling2D, Dropout, Flatten, Dense\n",
        "\n",
        "(train_images, train_labels), (test_images, test_labels) = keras.datasets.mnist.load_data()\n",
        "print(len(train_images), \"train images\")\n",
        "print(len(test_images), \"test images\")"
      ],
      "metadata": {
        "id": "YM2cQ-w_WF-w"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's take a look at an image to see what we're working with. We'll also print the size of each image and the max value in each image array."
      ],
      "metadata": {
        "id": "tZBUN_QpbqFu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"Size of each image =\", train_images[0].shape)\n",
        "print(\"Max value of each image =\", train_images[0].max())\n",
        "print()\n",
        "print(\"First train image is a\", train_labels[0])\n",
        "print(\"First train image:\")\n",
        "plt.imshow(train_images[0]);"
      ],
      "metadata": {
        "id": "PeNWtrzcZV9j"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's do some quick pre-processing of the data by dividing all values in each image by 255 (the max value) so that everything is normalized to 1. We also need to change the shape of each to 28x28x1 (rather than the current 28x28)."
      ],
      "metadata": {
        "id": "LgUoL0OmdltC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "train_images = train_images/255\n",
        "test_images = test_images/255\n",
        "\n",
        "train_images = train_images.reshape(len(train_images), 28, 28, 1)\n",
        "test_images = test_images.reshape(len(test_images), 28, 28, 1)"
      ],
      "metadata": {
        "id": "ESAc9vU_b0fy"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now it's time to define our model. Let's go with the following order:\n",
        "\n",
        "- a 2D convolution layer with 32 filters and a filter (kernel) size of 3x3, applying a ReLU activation function\n",
        "- a 2D convolution layer with 64 filters and a filter size of 3x3, applying a ReLU activation function\n",
        "- a max pooling layer (pools according to the maximum value in the window) with a window size of 2x2\n",
        "- a dropout layer (randomly activates and deactivates some neurons to remove biases)\n",
        "- a dense layer with 128 neurons and a ReLU activation function\n",
        "- another dropout layer\n",
        "- a final dense layer with 10 neurons and a softmax activation function -- this will return the probability of each of the 10 labels (note that softmax activation functions are common when returning probabilities)"
      ],
      "metadata": {
        "id": "eK8hPVYKfABf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model = keras.Sequential()\n",
        "# 32 convolution filters used each of size 3x3\n",
        "model.add(Conv2D(32, kernel_size=(3, 3), activation='relu', input_shape=(28, 28, 1)))\n",
        "# 64 convolution filters used each of size 3x3\n",
        "model.add(Conv2D(64, (3, 3), activation='relu'))\n",
        "# choose the best features via pooling\n",
        "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
        "# randomly turn neurons on and off to improve convergence\n",
        "model.add(Dropout(0.25))\n",
        "# flatten since too many dimensions, we only want a classification output\n",
        "model.add(Flatten())\n",
        "# fully connected to get all relevant data\n",
        "model.add(Dense(128, activation='relu'))\n",
        "# one more dropout\n",
        "model.add(Dropout(0.5))\n",
        "# output a softmax to squash the matrix into output probabilities\n",
        "model.add(Dense(10, activation='softmax'))"
      ],
      "metadata": {
        "id": "PpyFFi5aeFdE"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now we need to define some things that will determine how the model does the calculation while training. Here, we are defining our optimizer algorithm, our loss function, and the metric on which we are training."
      ],
      "metadata": {
        "id": "LO6_LltpiQ6Y"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model.compile(optimizer=tf.compat.v1.train.AdamOptimizer(),\n",
        "              loss='sparse_categorical_crossentropy',\n",
        "              metrics=['accuracy'])"
      ],
      "metadata": {
        "id": "gzvSvRb1hp3a"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now we can train our model on our training images and labels. We will set the number of epochs to 5, meaning the algorithm will iteratively train 5 times, using the results of the previous iteration to inform the current one."
      ],
      "metadata": {
        "id": "yR5rwDDYi22J"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model.fit(train_images, train_labels, epochs=5)"
      ],
      "metadata": {
        "id": "xJYNEKL6igEg"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The model is trained! Let's now assess the accuracy of the model on the *test* data (which the model has not yet seen)."
      ],
      "metadata": {
        "id": "fEKwvv-2jMkI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "test_loss, test_acc = model.evaluate(test_images, test_labels)\n",
        "\n",
        "print('Test accuracy =', test_acc)"
      ],
      "metadata": {
        "id": "nA4VL7qwjEHH"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We achieve a pretty good accuracy of ~99%! Let's now make some predictions."
      ],
      "metadata": {
        "id": "9XKoKgapjcLy"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# select some random test images\n",
        "random_idcs = np.random.randint(10000, size=10)\n",
        "# make predictions using the models\n",
        "predictions = model.predict(test_images[random_idcs])\n",
        "\n",
        "# plot a few of them\n",
        "for i,idx in enumerate(random_idcs):\n",
        "    print(\"True label is\", test_labels[idx])\n",
        "\n",
        "    fig, ax = plt.subplots(1, 2, figsize=(10,5))\n",
        "\n",
        "    ax[0].imshow(test_images[idx])\n",
        "    ax[1].bar(np.arange(10), predictions[i])\n",
        "    ax[1].set_xlabel(\"Predicted Label\")\n",
        "    ax[1].set_ylabel(\"Probability\")\n",
        "    ax[1].set_xticks(np.arange(10))\n",
        "    plt.show()\n",
        "\n",
        "    print()"
      ],
      "metadata": {
        "id": "y3QQP84Tjb0Q"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "I'll let you give it a shot and let you design your own CNN with your own layers. Use the model from the cell above and change the layers and hyperparameters to whatever you want, but leave the first and last layers the same. If you want a suggestion, try the following:\n",
        "\n",
        "- a 2D convulution cell (first layer)\n",
        "- a max pooling layer\n",
        "- a 2D convolution layer\n",
        "- a max pooling layer\n",
        "- a dense layer\n",
        "- a dropout layer\n",
        "- a dense layer with 10 neurons (last layer)\n",
        "\n",
        "How does the accuracy of this model compare to the one we trained before?"
      ],
      "metadata": {
        "id": "SgC5h-9ukhGi"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "1ZqRYaKMlRPk"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Additional resources\n",
        "\n",
        "Deep learning algorithms, including convolutional neural networks, are commonly used in physics and astronomy (and many other fields) when faced with large data sets that are impossible to parse by-eye. Many of these projects are hosted on [Zooniverse](https://www.zooniverse.org/) - I encourage you to check them out!"
      ],
      "metadata": {
        "id": "CvHXwz7gsKLD"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Acknoledgements\n",
        "\n",
        "This exercise is inspired by the work of the [TensorFlow team](https://www.tensorflow.org)."
      ],
      "metadata": {
        "id": "c-6_dUQtjzKX"
      }
    }
  ]
}