Spring-Symposium-Python-I.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Science Tutorial\n",
    "\n",
    "Sign in here: https://goo.gl/forms/SFsCpO4Vy7PDa2Rg2\n",
    "\n",
    "Now that we've covered some Python basics, we will begin a tutorial going through many tasks a data scientist may perform.  We will obtain real world data and go through the process of auditing, analyzing, visualing, and building classifiers from the data.\n",
    "\n",
    "We will use a database of selected disease statistics of various contries from the Global Health Observatory. The data is organized by country and year, with the number of specific incidents of each disease listed. The attributes and domain of each entry are described by the table below:\n",
    "\n",
    "| Attribute                     | Domain                          |\n",
    "|-------------------------------|---------------------------------|\n",
    "| 1. Country                    | String                          |\n",
    "| 2. Year                       | Year (2009-2014)                |\n",
    "| 3. T.b. gambiense             | Integer                         |\n",
    "| 4. T.b. rhodesiense           | Integer                         |\n",
    "| 5. Cholera                    | Integer                         |\n",
    "| 6. Meningitis (suspected)     | Integer                         |\n",
    "| 7. Congenital Rubella         | Integer                         |\n",
    "| 8. Diphtheria                 | Integer                         |\n",
    "| 9. Japanese encephalitis      | Integer                         |\n",
    "| 10. Leprosy                   | Integer                         |\n",
    "| 11. Malaria                   | Integer                         |\n",
    "| 12. Measles                   | Integer                         |\n",
    "| 13. Mumps                     | Integer                         |\n",
    "| 14. Neonatal Tetanus          | Integer                         |\n",
    "| 15. Pertussis                 | Integer                         |\n",
    "| 16. Plague                    | Integer                         |\n",
    "| 17. Poliomyelitis             | Integer                         |\n",
    "| 18. Rubella                   | Integer                         |\n",
    "| 19. Total Tetanus             | Integer                         |\n",
    "| 20. Tuberculosis              | Integer                         |\n",
    "| 21. Yellow Fever              | Integer                         |\n",
    "| 22. Cutaneous Leishmaniasis   | Integer                         |\n",
    "| 23. Visceral Leishmaniasis    | Integer                         |\n",
    "\n",
    "For more information on this data set:\n",
    "http://apps.who.int/gho/data/node.home\n",
    "\n",
    "## Obtaining the Data\n",
    "Lets begin by programmatically obtaining the data.  Here I'll define a function we can use to make HTTP requests and download the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#define download_file function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll specify the url of the file and the file name we will save to"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#specify url and filename"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And make a call to <code>download_file</code>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#call our download_file function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:**  If you see an InsecurePlatformWarning message, ignore it. More info can be found here: https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning\n",
    "\n",
    "Now this might seem like overkill for downloading a single, small csv file, but we can use this same function to access countless APIs available on the World Wide Web by building an API request in the url.\n",
    "\n",
    "## Wrangling the Data\n",
    "Now that we have some data, lets get it into a useful form.  For this task we will use a package called pandas. pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for Python.  The most fundamental data structure in pandas is the dataframe, which is similar to the data.frame data structure found in the R statistical programming language.\n",
    "\n",
    "For more information: http://pandas.pydata.org\n",
    "\n",
    "pandas dataframes are a 2-dimensional labeled data structures with columns of potentially different types.  Dataframes can be thought of as similar to a spreadsheet or SQL table.\n",
    "\n",
    "There are numerous ways to build a dataframe with pandas.  Since we have already attained a csv file, we can use a parser built into pandas called <code>read_csv</code> which will read the contents of a csv file directly into a data frame.\n",
    "\n",
    "For more information: http://pandas.pydata.org/pandas-docs/dev/generated/pandas.io.parsers.read_csv.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#import pandas and load dataset into a frame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets take a look at some simple statistics for the **Cholera** column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#describe cholera column"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Referring to the documentation, the data contains 1164 entries. However, if we take a look at the \"count\" section, it shows only 245 entries. This is because the original data is filled with empty strings, which pandas automatically converts to Numpy's <code>nan</code> datatype, or \"Not a Number\". \n",
    "\n",
    "Lets take a look at another column, this time **Pertussis**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#describe pertussis column"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well at least the name is correct.  We were expecting a mean and standard deviation, and now the data type is an object.  \n",
    "\n",
    "Whats up with our data?\n",
    "\n",
    "We have arrived at arguably the most important part of performing data science: dealing with messy data.  One of most important tools in a data scientist's toolbox is the ability to audit, clean, and reshape data.  The real world is full of messy data and your sources may not always have data in the exact format you desire.\n",
    "\n",
    "In this case we are working with csv data, which is a relatively straightforward format, but this will not always be the case when performing real world data science.  Data comes in all varieties from csv all the way to something as unstructured as a collection of emails or documents.  A data scientist must be versed in a wide variety of technologies and methodologies in order to be successful.\n",
    "\n",
    "Now, lets do a little bit of digging into why were are not getting a numeric pandas column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#find unique values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using <code>unique</code> we can see that '0 0', '5 5', and '2 2' all appear as distinct values in this series. Because of the space between the numbers, Python has classified these as *strings* rather than *integers*. Indeed, it's not immediately obvious that these were meant to be legitimate entries in the first place.\n",
    "\n",
    "Lets see what we can do with these unrecognized values. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#convert column to numeric values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we have attempted to convert the **Pertussis** series to a numeric type.  Lets see what the unique values are now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#find new unique values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The decimal point after each number means that it is an integer value being represented by a floating point number.  Now instead of our pesky *strings* we have <code>nan</code> (not a number).  <code>nan</code> is a construct used by pandas to represent the absence of value.  It is a data type that comes from the package numpy, used internally by pandas, and is not part of the standard Python library.\n",
    "\n",
    "Now that we have <code>nan</code> values in place of strings, we can use some nice features in pandas to deal with these missing values.\n",
    "\n",
    "What we are about to do is what is called \"imputing\" or providing a replacement for missing values so the data set becomes easier to work with.  There are a number of strategies for imputing missing values, all with their own pitfalls.  In general, imputation introduces some degree of bias to the data, so the imputation strategy taken should be in an attempt to minimize that bias.\n",
    "\n",
    "Here, we will simply ignore all of the <code>nan</code> values, however other strategies such as replacing the <code>nan</code>'s with the mean of the data are also commonly used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#convert whole data frame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<code>health_data.mean().round()</code> will take the mean of each column (this computation ignores the currently present nan values), then round, and return a dataframe indexed by the columns of the original dataframe.\n",
    "\n",
    "This function can be used to replace all missing values with the mean of each column. In this tutorial however, we will not use this method, because the large number of missing values would greatly skew our standard deviations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#find mean values for imputing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have figured out how to impute these missing values, lets start over and quickly apply this technique to the entire dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#quickly load and convert data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#check unique values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Structurally, Pandas dataframes are a collection of Series objects sharing a common index.  In general, the Series object and Dataframe object share a large number of functions with some behavioral differences.  In other words, whatever computation you can do on a single column can generally be applied to the entire dataframe.\n",
    "\n",
    "Now we can use the dataframe version of <code>describe</code> to get an overview of all of our data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#overview description of data frame"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}