03_bees_vs_wasps.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Practicum AI Logo image](images/practicum_ai_logo.png) <img src='images/practicumai_deep_learning.png' alt='Practicum AI: Deep Learning Foundations icon' align='right' width=50>\n",
    "\n",
    "***\n",
    "# Hyperparameter Optimization\n",
    "\n",
    "Amelia's nutrition study is going so well that her colleagues have heard about her growing AI skills! An entomologist colleague reached out to Amelia for help with their project. They would like to classify images as being a bee, a wasp, some other insect, or none of these.\n",
    "\n",
    "Luckily, the entomologist colleague knows about the popular online repository of datasets, Kaggle, and already found a great dataset of images to use! [Check out the dataset information](https://www.kaggle.com/datasets/jerzydziewierz/bee-vs-wasp). [![Image of bees and wasps from the dataset cover image](images/bees_wasps_dataset-cover.png)]((https://www.kaggle.com/datasets/jerzydziewierz/bee-vs-wasp))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Import the libraries we will use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf   # Import the TensorFlow library, which provides tools for deep learning.\n",
    "import pandas as pd  # Import the pandas library, used for data manipulation and analysis.\n",
    "\n",
    "# Used for data management\n",
    "import requests\n",
    "import os\n",
    "import time\n",
    "import tarfile\n",
    "\n",
    "import matplotlib.pyplot as plt  # Import the matplotlib library for plotting and visualization.\n",
    "# This line allows for the display of plots directly within the Jupyter notebook interface.\n",
    "%matplotlib inline  \n",
    " \n",
    "# Import Keras libraries\n",
    "from tensorflow.keras.models import Sequential  # Import the Sequential model: a linear stack of layers from Keras module in TensorFlow.\n",
    "from tensorflow.keras.layers import Dense  # Import the Dense layer: a fully connected neural network layer from Keras module in TensorFlow.\n",
    "from tensorflow.keras.layers import Flatten  # Import the Flatten layer: used to convert input data into a 1D array from Keras module in TensorFlow.\n",
    "from tensorflow.keras.losses import SparseCategoricalCrossentropy  # Import the SparseCategoricalCrossentropy loss function from Keras module in TensorFlow.\n",
    "from tensorflow.keras import layers \n",
    "from tensorflow.keras import losses\n",
    "from sklearn.metrics import confusion_matrix \n",
    "import numpy as np "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Getting the data\n",
    "\n",
    "Even once we have identified the dataset we want to use, getting it can be a challenge. Many AI datasets are large, require authentication even for free datasets, and may require some cleanup before working with them.\n",
    "\n",
    "You can download the data from Kaggle, but need a free account. Additional steps are also needed to get the data into a usable format.\n",
    "\n",
    "Git and GitHub.com are generally not well suited to large files (GitHub's limit is generally about 100Mb per file). And if we add all the images individually to the repository, the about 20,000 image files make operations take a long time.\n",
    "\n",
    "If you are doing this as part of a workshop, we will provide the path to the data. \n",
    "\n",
    "We do have the dataset [hosted for download from Dropbox here as a `tar.gz` file](https://www.dropbox.com/s/x70hm8mxqhe7fa6/bee_vs_wasp.tar.gz?dl=0) that you can download and should be ready to use. \n",
    "\n",
    "### If needed, download the dataset\n",
    "\n",
    "The following code block is quite large. **You do not need to understand everything!** This block will look for the data files required for this notebook in some common locations. If it can't find the data, it will ask if you know where it is. If you do, answer yes and provide the path to the data (up to and including the `bee_vs_wasp` folder name). If not, it will ask if you want to download it. If you answer yes, it will download the data and extract it into your data folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def download_file(url=\"https://www.dropbox.com/s/x70hm8mxqhe7fa6/bee_vs_wasp.tar.gz?dl=1\", filename=\"bee_vs_wasp.tar.gz\"):\n",
    "\n",
    "    # Download the file using requests\n",
    "    response = requests.get(url, stream=True)\n",
    "\n",
    "    # Create a file object and write the response content in chunks\n",
    "    with open(filename, \"wb\") as f:\n",
    "        for chunk in response.iter_content(chunk_size=8192):\n",
    "            f.write(chunk)\n",
    "\n",
    "    # Wait for the file to finish downloading\n",
    "    while not os.path.exists(filename):\n",
    "        time.sleep(1)\n",
    "\n",
    "    # Print a success message\n",
    "    print(f\"Downloaded {filename} successfully.\")\n",
    "\n",
    "def extract_file(filename, data_folder):\n",
    "    # Check if the file is a tar file\n",
    "    if tarfile.is_tarfile(filename):\n",
    "        # Open the tar file\n",
    "        tar = tarfile.open(filename, \"r:gz\")\n",
    "        # Extract all the files to the data folder\n",
    "        tar.extractall(data_folder)\n",
    "        # Close the tar file\n",
    "        tar.close()\n",
    "        # Print a success message\n",
    "        print(f\"Extracted {filename} to {data_folder} successfully.\")\n",
    "    else:\n",
    "        # Print an error message\n",
    "        print(f\"{filename} is not a valid tar file.\")\n",
    "    \n",
    "def manage_data(folder_name='bee_vs_wasp'):\n",
    "\n",
    "    # Check common paths of where the data might be on different systems\n",
    "    likely_paths= [os.path.normpath(f'/blue/practicum-ai/share/data/{folder_name}'),\n",
    "                   os.path.normpath(f'/project/scinet_workshop2/data/{folder_name}'),\n",
    "                   os.path.join('data', folder_name),\n",
    "                   os.path.normpath(folder_name)]\n",
    "    \n",
    "    for path in likely_paths:\n",
    "        if os.path.exists(path):\n",
    "            print(f'Found data at {path}.')\n",
    "            return path\n",
    "\n",
    "    answer = input(f'Could not find data in the common locations. Do you know the path? (yes/no): ')\n",
    "\n",
    "    if answer.lower() == 'yes':\n",
    "        path = os.path.join(os.path.normpath(input('Please enter the path to the data folder: ')),folder_name)\n",
    "        if os.path.exists(path):\n",
    "            print(f'Thanks! Found your data at {path}.')\n",
    "            return path\n",
    "        else:\n",
    "            print(f'Sorry, that path does not exist.')\n",
    "    \n",
    "    answer = input('Do you want to download the data? (yes/no): ')\n",
    "\n",
    "    if answer.lower() == 'yes':\n",
    "    \n",
    "        ''' Check and see if the downloaded data is inside the .gitignore file, and adds them to the list of files to ignore if not. \n",
    "        This is to prevent the data from being uploaded to the repository, as the files are too large for GitHub.'''\n",
    "        \n",
    "        if os.path.exists('.gitignore'):\n",
    "            with open('.gitignore', 'r') as f:\n",
    "                ignore = f.read().split('\\n')\n",
    "        # If the .gitignore file does not exist, create a new one\n",
    "        elif not os.path.exists('.gitignore'):\n",
    "            with open('.gitignore', 'w') as f:\n",
    "                f.write('')\n",
    "            ignore = []\n",
    "        else:\n",
    "            ignore = []\n",
    "\n",
    "        # Check if the .gz file is in the ignore list\n",
    "        if 'bee_vs_wasp.tar.gz' not in ignore:\n",
    "            ignore.append('bee_vs_wasp.tar.gz')\n",
    "            \n",
    "        # Check if the data/ folder is in the ignore list\n",
    "        if 'data/' not in ignore:\n",
    "            ignore.append('data/')\n",
    "\n",
    "        # Write the updated ignore list back to the .gitignore file\n",
    "        with open('.gitignore', 'w') as f:\n",
    "            f.write('\\n'.join(ignore))\n",
    "\n",
    "        print(\"Updated .gitignore file.\")\n",
    "        print('Downloading data, this may take a minute.')\n",
    "        download_file()\n",
    "        print('Data downloaded, unpacking')\n",
    "        extract_file(\"bee_vs_wasp.tar.gz\", \"data\")\n",
    "        print('Data downloaded and unpacked. Now available at data/bee_vs_wasp.')\n",
    "        return os.path.normpath('data/bee_vs_wasp')   \n",
    "\n",
    "    print('Sorry, I cannot find the data. Please download it manually from https://www.dropbox.com/s/x70hm8mxqhe7fa6/bee_vs_wasp.tar.gz and unpack it to the data folder.')      \n",
    "\n",
    "\n",
    "data_path = manage_data()    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Examine some images\n",
    "\n",
    "Many of the steps in this notebook are written as functions, making it easier to run these steps repeatedly as you work on optimizing the various hyperparameters.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# The data path should be set from the cell above. \n",
    "# If that failed and you want to set it manually, use the line below.\n",
    "# data_path = \"data/bee_vs_wasp\"\n",
    "\n",
    "\n",
    "def load_display_data(path, batch_size=32, shape=(80,80,3), show_pictures=True):\n",
    "    '''Takes a path, batch size, target shape for images and optionally whether to show sample images.\n",
    "       Returns training and testing datasets\n",
    "    '''\n",
    "    print(\"***********************************************************************\")\n",
    "    print(\"Load data:\")\n",
    "    print(f\"  - Loading the dataset from: {path}.\")\n",
    "    print(f\"  - Using a batch size of: {batch_size}.\")\n",
    "    print(f\"  - Resizing input images to: {shape}.\")\n",
    "    print(\"***********************************************************************\")\n",
    "    # Define the directory path\n",
    "    directory_path = path\n",
    "    \n",
    "    # Define the batch size\n",
    "    batch_size = batch_size\n",
    "    \n",
    "    # Define the image size using the 1st 2 elements of the shape parameter\n",
    "    # We don't need the number of channels here, just the dimensions to use\n",
    "    image_size = shape[:2]\n",
    "    \n",
    "    # Load the dataset\n",
    "    X_train = tf.keras.preprocessing.image_dataset_from_directory(\n",
    "        directory_path,\n",
    "        batch_size=batch_size,\n",
    "        image_size=image_size,\n",
    "        validation_split=0.2,\n",
    "        subset='training',\n",
    "        seed=123,\n",
    "        labels='inferred',\n",
    "        label_mode='int'\n",
    "    )\n",
    "    \n",
    "    X_test = tf.keras.preprocessing.image_dataset_from_directory(\n",
    "        directory_path,\n",
    "        batch_size=batch_size,\n",
    "        image_size=image_size,\n",
    "        validation_split=0.2,\n",
    "        subset='validation',\n",
    "        seed=123,\n",
    "        labels='inferred',\n",
    "        label_mode='int'\n",
    "    )\n",
    "\n",
    "    if show_pictures:\n",
    "        # Get the class names\n",
    "        class_names = X_train.class_names\n",
    "        print(class_names)\n",
    "\n",
    "        # Display up to 3 images from each of the categories\n",
    "        for i, class_name in enumerate(class_names):\n",
    "            plt.figure(figsize=(10, 10))\n",
    "            for images, labels in X_train.take(2):\n",
    "                images = images.numpy()\n",
    "                labels = labels.numpy()\n",
    "\n",
    "                # Filter images of the current class\n",
    "                class_images = images[labels == i]\n",
    "                \n",
    "                # Number of images to show.\n",
    "                # Limited by number of this class in the batch or specific number\n",
    "                num_images = min(len(class_images), 3)\n",
    "                \n",
    "                for j in range(num_images):\n",
    "                    ax = plt.subplot(1, num_images, j + 1)\n",
    "                    plt.imshow(class_images[j].astype(\"uint8\"))\n",
    "                    plt.title(class_name)\n",
    "                    plt.axis(\"off\")\n",
    "            plt.show()\n",
    "    return X_train, X_test\n",
    "\n",
    "X_train, X_test = load_display_data(data_path, batch_size=32, shape=(80,80,3), show_pictures=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Make our model\n",
    "\n",
    "This function creates the model we will use.\n",
    "\n",
    "One hyperparameter to explore is the activation function, which is set when making the model. We start with a ReLU as the default, but you can try others. For simplicity, we will use the same activation function for all but the last layer of the model, but you could change them individually.\n",
    "\n",
    "The last layer will almost always use a Softmax, which makes all the output values between 0 and 1 and sum to 1, transforming them into probabilities of the input belonging to each possible class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def make_model(activation='relu', shape=(80,80,3), num_classes=4):\n",
    "    '''Sets up a model. \n",
    "          Takes in an activation function, shape for the input images, and number of classes.\n",
    "          Returns the model.\n",
    "    '''\n",
    "    print(\"***********************************************************************\")\n",
    "    print(\"Make model:\")\n",
    "    print(f\"  - Using the activation function: {activation}.\")\n",
    "    print(f\"  - Model will have {num_classes} classes.\")\n",
    "    print(\"***********************************************************************\")\n",
    "    \n",
    "    # Define the model\n",
    "    model = tf.keras.Sequential([\n",
    "        layers.Conv2D(32, (3, 3), activation=activation, input_shape=shape),\n",
    "        layers.MaxPooling2D((2, 2)),\n",
    "        layers.Conv2D(64, (3, 3), activation=activation),\n",
    "        layers.MaxPooling2D((2, 2)),\n",
    "        layers.Conv2D(128, (3, 3), activation=activation),\n",
    "        layers.MaxPooling2D((2, 2)),\n",
    "        layers.Flatten(),\n",
    "        layers.Dense(128, activation=activation),\n",
    "        layers.Dense(num_classes, activation='softmax')\n",
    "    ])\n",
    "\n",
    "    return model\n",
    "\n",
    "model = make_model()\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Compile and train the model\n",
    "\n",
    "This step compiles the model, getting it ready for training. The primary hyperparameters here are:\n",
    "* the **loss function** (how we determine how close the predicted output is from the known output values),\n",
    "* the **optimization function** (how we determine what parameters should be updated and how),\n",
    "* the **learning rate** (how much each parameter should be adjusted), \n",
    "* and how many **epochs** should be run (remember, an epoch is a full pass through all the training data). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "\n",
    "# Define a function that takes an optimizer name as a string\n",
    "def load_optimizer(optimizer_name):\n",
    "  # Check if the optimizer name is valid\n",
    "  if optimizer_name in tf.keras.optimizers.__dict__:\n",
    "    # Return the corresponding optimizer function\n",
    "    return tf.keras.optimizers.__dict__[optimizer_name]\n",
    "  else:\n",
    "    # Raise an exception if the optimizer name is invalid\n",
    "    raise ValueError(f\"Invalid optimizer name: {optimizer_name}\")\n",
    "\n",
    "def compile_train_model(X_train, X_test, model,\n",
    "                        loss=SparseCategoricalCrossentropy(from_logits=True),\n",
    "                        optimizer='Adam', learning_rate=0.0001, epochs=10):\n",
    "    '''Compiles and trains the model. \n",
    "          Takes in an X_train, X_test, model, loss function, optimizer, learning rate,\n",
    "          and epochs.\n",
    "          Returns the compiled model and training history.\n",
    "    '''\n",
    "    print(\"***********************************************************************\")\n",
    "    print(\"Compile and Train the model:\")\n",
    "    print(f\"  - Using the loss funtion: {loss}.\")\n",
    "    print(f\"  - Using the optimizer: {optimizer}.\")\n",
    "    print(f\"  - Using learning rate of: {learning_rate}.\")\n",
    "    print(f\"  - Running for {epochs} epochs.\")\n",
    "    print(\"***********************************************************************\")\n",
    "    # Compile the model\n",
    "    \n",
    "    opt = load_optimizer(optimizer)(learning_rate=learning_rate)\n",
    "    \n",
    "    model.compile(optimizer=opt,\n",
    "                  loss=loss,\n",
    "                  metrics=['accuracy'])\n",
    "    # Train the model\n",
    "    history = model.fit(X_train, epochs=epochs, validation_data=X_test)\n",
    "    \n",
    "    return model, history\n",
    "\n",
    "model, history = compile_train_model(X_train, X_test, model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Evaluate the model\n",
    "\n",
    "Now that we have trained our model, let's evaluate how it does.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def evaluate_model(X_train, X_test, model, history, num_classes=4):\n",
    "    '''Evaluates a model. \n",
    "          Takes in an X_train, X_test, model, history, number of classes.\n",
    "    '''\n",
    "    print(\"***********************************************************************\")\n",
    "    print(\"Evaluate the model:\")\n",
    "    print(\"***********************************************************************\")\n",
    "    # Evaluate the model\n",
    "    loss, accuracy = model.evaluate(X_test)\n",
    "    print(f'Test loss: {loss}')\n",
    "    print(f'Test accuracy: {accuracy}')\n",
    "\n",
    "\n",
    "    # Plot the training and validation loss over time\n",
    "    plt.plot(history.history['loss'], label='Training Loss')\n",
    "    plt.plot(history.history['val_loss'], label='Validation Loss')\n",
    "    plt.title('Training and Validation Loss over Time')\n",
    "    plt.xlabel('Epoch')\n",
    "    plt.ylabel('Loss')\n",
    "    plt.legend()\n",
    "    plt.show()\n",
    "    \n",
    "    # Plot the training and validation accuracy over time\n",
    "    plt.plot(history.history['accuracy'], label='Training Accuracy')\n",
    "    plt.plot(history.history['val_accuracy'], label='Validation Accuracy')\n",
    "    plt.title('Training and Validation Accuracy over Time')\n",
    "    plt.xlabel('Epoch')\n",
    "    plt.ylabel('Accuracy')\n",
    "    plt.legend()\n",
    "    plt.show()\n",
    "\n",
    "    # Get the class names\n",
    "    class_names = X_test.class_names\n",
    "\n",
    "    # Make predictions on the test set\n",
    "    y_pred = np.argmax(model.predict(X_test), axis=-1)\n",
    "    \n",
    "    # Get the true labels\n",
    "    y_true = np.concatenate([y for x, y in X_test], axis=0)\n",
    "    \n",
    "    # Compute the confusion matrix\n",
    "    cm = confusion_matrix(y_true, y_pred)\n",
    "    \n",
    "    # Plot the confusion matrix\n",
    "    plt.imshow(cm, cmap=plt.cm.Blues)\n",
    "    plt.title('Confusion Matrix')\n",
    "    plt.xlabel('Predicted Label')\n",
    "    plt.ylabel('True Label')\n",
    "    plt.xticks(range(num_classes),class_names)\n",
    "    plt.yticks(range(num_classes), class_names)\n",
    "    plt.colorbar()\n",
    "    for i in range(num_classes):\n",
    "        for j in range(num_classes):\n",
    "            plt.text(j, i, cm[i, j], ha='center', va='center', color='black')\n",
    "    plt.show()\n",
    "\n",
    "evaluate_model(X_train, X_test, model, history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Explore hyperparameters!\n",
    "\n",
    "OK, we've trained the model once using some decent first guesses. As you can see in the Training and Validation Accuracy over Time graph above, the model's accuracy improved against the training data, but it's validation accuracy stayed pretty stagnant. What do you think that says about the model? Do you think the model is overfitting, underfitting, or just right? If you're not sure, you can refer back to Module 3, How to Tune Your Model, for a refresher.\n",
    "\n",
    "In the Confusion Matrix above, we can see that the model is doing a good job of classifying wasps, but not so well with other insects. Why do you think that is? What could we do to improve the model's performance?\n",
    "\n",
    "Now, we can see if we can do better by exploring different hyperparameters.\n",
    "\n",
    "While there are methods to explore different hyperparameters systematically and track the results more efficiently, we will rely on some ad-hoc exploration and keep everything in the notebook.\n",
    "\n",
    "The following function pulls all the steps from above into a single function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def the_whole_shebang(path, batch_size, shape, classes, activation, loss, optimizer, show_pictures=True):\n",
    "    \n",
    "    X_train, X_test = load_display_data(path, batch_size, shape, show_pictures)\n",
    "    model = make_model(activation=activation, shape=shape, num_classes=classes)\n",
    "    model, history = compile_train_model(X_train, X_test, model, loss=loss,\n",
    "                        optimizer=optimizer, learning_rate=learning_rate, epochs=epochs)\n",
    "    evaluate_model(X_train, X_test, model, history, classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Copy the next cell and change hyperparameters\n",
    "\n",
    "You can copy the next cell multiple times and adjust the hyperparameters to compare results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#data_path = 'data/bee_vs_wasp/' # Path to the data.\n",
    "        # This is defined above, only need to change if you change datasets\n",
    "    \n",
    "show_pictures = True # Show sample images from the dataset? Keep on at first, but may become distracting.\n",
    "                     # Set to False to turn off\n",
    "# Hyperparameters\n",
    "shape = (80,80,3)  # Dimensions to use for the images...the raw data are 80x80\n",
    "                   #  color images, but you could down-sample them\n",
    "                   #  or convert them to black and white if you wanted\n",
    "batch_size = 32  # What batch size to use\n",
    "classes = 4 # We have 4 classes in our dataset: bee, wasp, other_insect, other_noninsect\n",
    "            # Only change this if you change the dataset\n",
    "activation='relu' # The activation function is an important hyperparameter\n",
    "                  # Other activations functions to try: tanh, sigmoid\n",
    "\n",
    "loss = SparseCategoricalCrossentropy(from_logits=True) # Loss function\n",
    "        # Other loss functions to try: losses.CategoricalHinge()\n",
    "        #                              losses.KLDivergence()\n",
    "\n",
    "optimizer = 'Adagrad' # Optimizer: others to try: Adam, RMSprop, Adagrad \n",
    "\n",
    "learning_rate = 0.001 # Try increasing or decreasing the learning rate by an order of magnitude\n",
    "\n",
    "epochs = 10 # Try running more epochs\n",
    "\n",
    "# Run everything with these hyperparameters\n",
    "the_whole_shebang(data_path, batch_size, shape, classes, activation, loss, optimizer, show_pictures)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "## Push changes to GitHub <img src=\"images/push_to_github.png\" alt=\"Push to GitHub icon\" align=\"right\" width=150>\n",
    "\n",
    " Remember to **add**, **commit**, and **push** the changes you have made to this notebook to GitHub to keep your repository in sync.\n",
    "\n",
    "In Jupyter, those are done in the git tab on the left. In Google Colab, use File > Save a copy in GitHub."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}