C1_W3_Lab_2_custom-dense-layer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ungraded Lab: Building a Custom Dense Layer\n",
    "\n",
    "In this lab, we'll walk through how to create a custom layer that inherits the [Layer](https://keras.io/api/layers/base_layer/#layer-class) class. Unlike simple Lambda layers you did previously, the custom layer here will contain weights that can be updated during training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "DpioxwFXD9Is"
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    # %tensorflow_version only exists in Colab.\n",
    "    %tensorflow_version 2.x\n",
    "except Exception:\n",
    "    pass\n",
    "\n",
    "import tensorflow as tf\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Custom Layer with weights\n",
    "\n",
    "To make custom layer that is trainable, we need to define a class that inherits the [Layer](https://keras.io/api/layers/base_layer/#layer-class) base class from Keras. The Python syntax is shown below in the class declaration. This class requires three functions: `__init__()`, `build()` and `call()`. These ensure that our custom layer has a *state* and *computation* that can be accessed during training or inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jnVrzRT6BPWl"
   },
   "outputs": [],
   "source": [
    "# inherit from this base class\n",
    "from tensorflow.keras.layers import Layer\n",
    "\n",
    "class SimpleDense(Layer):\n",
    "\n",
    "    def __init__(self, units=32):\n",
    "        '''Initializes the instance attributes'''\n",
    "        super(SimpleDense, self).__init__()\n",
    "        self.units = units\n",
    "\n",
    "    def build(self, input_shape):\n",
    "        '''Create the state of the layer (weights)'''\n",
    "        # initialize the weights\n",
    "        w_init = tf.random_normal_initializer()\n",
    "        self.w = tf.Variable(name=\"kernel\",\n",
    "            initial_value=w_init(shape=(input_shape[-1], self.units),\n",
    "                                 dtype='float32'),\n",
    "            trainable=True)\n",
    "\n",
    "        # initialize the biases\n",
    "        b_init = tf.zeros_initializer()\n",
    "        self.b = tf.Variable(name=\"bias\",\n",
    "            initial_value=b_init(shape=(self.units,), dtype='float32'),\n",
    "            trainable=True)\n",
    "\n",
    "    def call(self, inputs):\n",
    "        '''Defines the computation from inputs to outputs'''\n",
    "        return tf.matmul(inputs, self.w) + self.b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can use our custom layer like below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# declare an instance of the class\n",
    "my_dense = SimpleDense(units=1)\n",
    "\n",
    "# define an input and feed into the layer\n",
    "x = tf.ones((1, 1))\n",
    "y = my_dense(x)\n",
    "\n",
    "# parameters of the base Layer class like `variables` can be used\n",
    "print(my_dense.variables)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's then try using it in simple network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "WwTPJT4DkTyW"
   },
   "outputs": [],
   "source": [
    "# define the dataset\n",
    "xs = np.array([-1.0,  0.0, 1.0, 2.0, 3.0, 4.0], dtype=float)\n",
    "ys = np.array([-3.0, -1.0, 1.0, 3.0, 5.0, 7.0], dtype=float)\n",
    "\n",
    "\n",
    "# use the Sequential API to build a model with our custom layer\n",
    "my_layer = SimpleDense(units=1)\n",
    "model = tf.keras.Sequential([my_layer])\n",
    "\n",
    "# configure and train the model\n",
    "model.compile(optimizer='sgd', loss='mean_squared_error')\n",
    "model.fit(xs, ys, epochs=500,verbose=0)\n",
    "\n",
    "# perform inference\n",
    "print(model.predict([10.0]))\n",
    "\n",
    "# see the updated state of the variables\n",
    "print(my_layer.variables)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "authorship_tag": "ABX9TyMm2GqP/jiEhAWgS1uK97vE",
   "collapsed_sections": [],
   "include_colab_link": true,
   "name": "CustomDenseLayer.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}