ch08-analysis-and-viz.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analysis and Visualization\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing Data\n",
    "\n",
    "<img src=\"images/analysis_fig.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading Data\n",
    "\n",
    "Let's try loading data into various analysis-ready forms using four tools in the Python ecosystem:\n",
    "- NumPy\n",
    "- PyTables\n",
    "- Pandas\n",
    "- Blaze"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading Data with NumPy\n",
    "\n",
    "Due to their power, many solutions involve NumPy arrays. NumPy arrays will be covered\n",
    "in detail in Chapter 9. For now, simply know that NumPy arrays are data structures\n",
    "for holding numbers. One easy way to transform a file into a NumPy array is\n",
    "with the `loadtxt()` function. Using this function, plain-text files holding columns of\n",
    "text delimited by commas, spaces, or tabs can be loaded directly into a NumPy array.\n",
    "The decay data in our earlier CSV-formatted example can be loaded directly into a\n",
    "NumPy array shown here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "decays_arr = np.loadtxt('data/decays.csv', delimiter=\",\", skiprows=1)\n",
    "decays_arr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Though not all were used in this example, many other\n",
    "options are available to customize the way a file is read with loadtxt(). To learn\n",
    "more about those, consult the numpy.loadtxt() documentation using the help command\n",
    "in IPython."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "help(np.loadtxt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading Data Into Pandas\n",
    "\n",
    "Loading data into a Pandas format is very straightforward. Note that the capability\n",
    "of numpy.loadtxt() can be repeated in Pandas with very similar syntax:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "decays_df = pd.read_csv('data/decays.csv') \n",
    "decays_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To write this data to an HDF5 file, we need to add\n",
    "just one line to the previous example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "hdf_df = decays_df.to_hdf('decays.h5', 'experimental')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading Data in Blaze\n",
    "\n",
    "Blaze is another Python tool capable of converting data from format to format. This\n",
    "tool is still in active development, but possesses impressive capabilities already."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import blaze as bz\n",
    "csv_data = bz.CSV('data/decays.csv')\n",
    "decays_tb = bz.Table(csv_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleaning and Munging Data\n",
    "\n",
    "### Missing Data\n",
    "\n",
    "Pandas is especially helpful in the event of missing data. In particular, Pandas has special\n",
    "methods for identifying, dropping, and replacing missing data.\n",
    "With only a few lines in IPython, the NaN rows from the previous data can be dropped\n",
    "from the dataset entirely:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "decay_df = pd.read_csv(\"data/many_decays.csv\")\n",
    "decay_df.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "decay_df.dropna()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# as in the previous example, load decays.csv into a NumPy array\n",
    "decaydata = np.loadtxt('data/decays.csv', delimiter=\",\", skiprows=1)\n",
    "\n",
    "# provide handles for the x and y columns\n",
    "time = decaydata[:,0]\n",
    "decays = decaydata[:,1]\n",
    "\n",
    "# import the matplotlib plotting functionality\n",
    "import matplotlib\n",
    "%matplotlib inline\n",
    "import pylab as plt\n",
    "\n",
    "plt.plot(time, decays)\n",
    "\n",
    "plt.xlabel('Time (s)')\n",
    "plt.ylabel('Decays')\n",
    "plt.title('Decays')\n",
    "plt.grid(True)\n",
    "#plt.savefig(\"decays_matplotlib.png\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Import various necessary Python and matplotlib packages\n",
    "import numpy as np\n",
    "import matplotlib.cm as cm\n",
    "from matplotlib.pyplot import figure, show, rc\n",
    "from matplotlib.patches import Ellipse\n",
    "\n",
    "# Create a square figure on which to place the plot\n",
    "fig = figure(figsize=(8,8))\n",
    "\n",
    "# Create square axes to hold the circular polar plot\n",
    "ax = fig.add_axes([0.1, 0.1, 0.8, 0.8], polar=True)\n",
    "\n",
    "# Generate 20 colored, angular wedges for the polar plot\n",
    "N = 20\n",
    "theta = np.arange(0.0, 2*np.pi, 2*np.pi/N)\n",
    "radii = 10*np.random.rand(N)\n",
    "width = np.pi/4*np.random.rand(N)\n",
    "bars = ax.bar(theta, radii, width=width, bottom=0.0)\n",
    "for r,bar in zip(radii, bars):\n",
    "    bar.set_facecolor(cm.jet(r/10.))\n",
    "    bar.set_alpha(0.5)\n",
    "\n",
    "# Using dictionaries, create a color scheme for the text boxes\n",
    "bbox_args = dict(boxstyle=\"round, pad=0.9\", fc=\"green\", alpha=0.5)\n",
    "bbox_white = dict(boxstyle=\"round, pad=0.9\", fc=\"1\", alpha=0.9)\n",
    "patch_white = dict(boxstyle=\"round, pad=1\", fc=\"1\", ec=\"1\")\n",
    "\n",
    "# Create various boxes with text annotations in them at specific\n",
    "# x and y coordinates\n",
    "ax.annotate(\" \",\n",
    "    xy=(.5,.93),\n",
    "    xycoords='figure fraction',\n",
    "    ha=\"center\", va=\"center\",\n",
    "    bbox=patch_white)\n",
    "\n",
    "ax.annotate('Matplotlib and the Python Ecosystem for Scientific Computing',\n",
    "    xy=(.5,.95),\n",
    "    xycoords='figure fraction',\n",
    "    xytext=(0, 0), textcoords='offset points',\n",
    "    size=15,\n",
    "    ha=\"center\", va=\"center\",\n",
    "    bbox=bbox_args)\n",
    "\n",
    "ax.annotate('Author and Lead Developer \\n of Matplotlib ',\n",
    "    xy=(.5,.82),\n",
    "    xycoords='figure fraction',\n",
    "    xytext=(0, 0), textcoords='offset points',\n",
    "    ha=\"center\", va=\"center\",\n",
    "    bbox=bbox_args)\n",
    "\n",
    "ax.annotate('John D. Hunter',\n",
    "    xy=(.5,.89),\n",
    "    xycoords='figure fraction',\n",
    "    xytext=(0, 0), textcoords='offset points',\n",
    "    size=15,\n",
    "    ha=\"center\", va=\"center\",\n",
    "    bbox=bbox_white)\n",
    "\n",
    "ax.annotate('Friday November 5th  \\n 2:00 pm \\n1106ME ',\n",
    "    xy=(.5,.25),\n",
    "    xycoords='figure fraction',\n",
    "    xytext=(0, 0), textcoords='offset points',\n",
    "    size=15,\n",
    "    ha=\"center\", va=\"center\",\n",
    "    bbox=bbox_args)\n",
    "\n",
    "ax.annotate('Sponsored by: \\n The Hacker Within, \\n'\n",
    "    'The University Lectures Committee, \\n The Department of '\n",
    "    'Medical Physics\\n and \\n The American Nuclear Society',\n",
    "    xy=(.78,.1),\n",
    "    xycoords='figure fraction',\n",
    "    xytext=(0, 0), textcoords='offset points',\n",
    "    size=9,\n",
    "    ha=\"center\", va=\"center\",\n",
    "    bbox=bbox_args)\n",
    "\n",
    "#fig.savefig(\"plot.pdf\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "# import the Bokeh plotting tools\n",
    "from bokeh import plotting as bp\n",
    "\n",
    "# as in the matplotlib example, load decays.csv into a NumPy array\n",
    "decaydata = np.loadtxt('data/decays.csv',delimiter=\",\",skiprows=1)\n",
    "\n",
    "# provide handles for the x and y columns\n",
    "time = decaydata[:,0]\n",
    "decays = decaydata[:,1]\n",
    "\n",
    "# define some output file metadata\n",
    "bp.output_file(\"decays.html\", title=\"Experiment 1 Radioactivity\")\n",
    "\n",
    "# create a figure with fun Internet-friendly features (optional)\n",
    "bp.figure(tools=\"pan,wheel_zoom,box_zoom,reset,previewsave\")\n",
    "\n",
    "# on that figure, create a line plot\n",
    "bp.line(time, decays, x_axis_label=\"Time (s)\", y_axis_label=\"Decays (#)\",\n",
    "     color='#1F78B4', legend='Decays per second')\n",
    "\n",
    "# additional customization to the figure can be specified separately\n",
    "bp.curplot().title = \"Decays\"\n",
    "bp.grid().grid_line_alpha=0.3\n",
    "\n",
    "# open a browser\n",
    "bp.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>\n",
       "<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>\n",
       "<link href='http://fonts.googleapis.com/css?family=Slabo%2027px' rel='stylesheet' type='text/css'>\n",
       "<style>\n",
       "\n",
       "/* #52575c <-grey\n",
       " * #f5f7fa <-nearly white\n",
       " * #62676c <-grey\n",
       " * #42474c <- also grey\n",
       " * #c2c7cc <-light grey\n",
       " * #f35858 <- salmon\n",
       " * #B73F98 <- o'reilly pink\n",
       " */\n",
       "\n",
       "@font-face {\n",
       "    font-family: \"Slabo 27px\", \"Helvetica Neue\", Helvetica, Arial, sans-serif;\n",
       "}\n",
       "\n",
       "\n",
       "#notebook_panel { /* main background */\n",
       "    background: rgb(245,245,245);\n",
       "}\n",
       "\n",
       "div.cell { /* set cell width */\n",
       "    width: 750px;\n",
       "}\n",
       "\n",
       "div #notebook { /* centre the content */\n",
       "    background: #fff; /* white background for content */\n",
       "    width: 1000px;\n",
       "    margin: auto;\n",
       "    padding-left: 0em;\n",
       "}\n",
       "\n",
       "#notebook li { /* More space between bullet points */\n",
       "margin-top:0.8em;\n",
       "}\n",
       "\n",
       "/* draw border around running cells */\n",
       "div.cell.border-box-sizing.code_cell.running { \n",
       "    border: 1px solid #111;\n",
       "}\n",
       "\n",
       "/* Put a solid color box around each cell and its output, visually linking them*/\n",
       "div.cell.code_cell {\n",
       "    background-color: rgb(256,256,256); \n",
       "    border-radius: 0px; \n",
       "    padding: 0.5em;\n",
       "    margin-left:1em;\n",
       "    margin-top: 1em;\n",
       "}\n",
       "\n",
       "\n",
       "div.text_cell_render{\n",
       "    font-family: \"Slabo 27px\", \"Helvetica Neue\", Helvetica, Arial, sans-serif;\n",
       "    line-height: 140%;\n",
       "    font-size: 125%;\n",
       "    font-weight: 400;\n",
       "    width:600px;\n",
       "    color:#42474c;\n",
       "    margin-left:auto;\n",
       "    margin-right:auto;\n",
       "}\n",
       "\n",
       "/* Formatting for header cells */\n",
       ".text_cell_render h1 {\n",
       "    font-family: \"Slabo 27px\", \"Helvetica Neue\", Helvetica, Arial, sans-serif;\n",
       "    font-style:regular;\n",
       "    font-weight: 200;    \n",
       "    font-size: 50pt;\n",
       "    line-height: 100%;\n",
       "    color:#B73F98;\n",
       "    margin-bottom: 0.5em;\n",
       "    margin-top: 0.5em;\n",
       "    display: block;\n",
       "}       \n",
       ".text_cell_render h2 {\n",
       "    font-family: 'Fenix', serif;\n",
       "    font-size: 22pt;\n",
       "    line-height: 100%;\n",
       "    margin-bottom: 0.1em;\n",
       "    margin-top: 0.3em;\n",
       "    display: block;\n",
       "}       \n",
       "\n",
       ".text_cell_render h3 {\n",
       "    font-family: 'Fenix', serif;\n",
       "    margin-top:12px;\n",
       "        font-size: 16pt;\n",
       "    margin-bottom: 3px;\n",
       "    font-style: regular;\n",
       "}\n",
       "\n",
       ".text_cell_render h4 {    /*Use this for captions*/\n",
       "    font-family: 'Fenix', serif;\n",
       "    font-size: 2pt;\n",
       "    text-align: center;\n",
       "    margin-top: 0em;\n",
       "    margin-bottom: 2em;\n",
       "    font-style: regular;\n",
       "}\n",
       "\n",
       ".text_cell_render h5 {  /*Use this for small titles*/\n",
       "    font-family: \"Slabo 27px\", \"Helvetica Neue\", Helvetica, Arial, sans-serif;\n",
       "    font-weight: 300;\n",
       "    font-size: 16pt;\n",
       "    color: #B73F98;\n",
       "    font-style: italic;\n",
       "    margin-bottom: .5em;\n",
       "    margin-top: 0.5em;\n",
       "    display: block;\n",
       "}\n",
       "\n",
       ".text_cell_render h6 { /*use this for copyright note*/\n",
       "    font-family: 'Source Code Pro', sans-serif;\n",
       "    font-weight: 300;\n",
       "    font-size: 9pt;\n",
       "    line-height: 100%;\n",
       "    color: grey;\n",
       "    margin-bottom: 1px;\n",
       "    margin-top: 1px;\n",
       "}\n",
       "\n",
       "    .CodeMirror{\n",
       "            font-family: \"Source Code Pro\";\n",
       "                        font-size: 90%;\n",
       "    }\n",
       "/*    .prompt{\n",
       "        display: None;\n",
       "    }*/\n",
       "        \n",
       "    \n",
       "    .warning{\n",
       "        color: rgb( 240, 20, 20 )\n",
       "        }  \n",
       "</style>\n",
       "<script>\n",
       "    MathJax.Hub.Config({\n",
       "                        TeX: {\n",
       "                           extensions: [\"AMSmath.js\"], \n",
       "                           equationNumbers: { autoNumber: \"AMS\", useLabelIds: true}\n",
       "                           },\n",
       "                tex2jax: {\n",
       "                    inlineMath: [ ['$','$'], [\"\\\\(\",\"\\\\)\"] ],\n",
       "                    displayMath: [ ['$$','$$'], [\"\\\\[\",\"\\\\]\"] ]\n",
       "                },\n",
       "                displayAlign: 'center', // Change this to 'center' to center equations.\n",
       "                \"HTML-CSS\": {\n",
       "                    styles: {'.MathJax_Display': {\"margin\": 4}}\n",
       "                }\n",
       "        });\n",
       "</script>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.core.display import HTML\n",
    "def css_styling():\n",
    "    styles = open(\"styles/custom.css\", \"r\").read()\n",
    "    return HTML(styles)\n",
    "css_styling()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}