02A-Getting-Started.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "eefcedb2-d95e-406f-adf0-a889c096010c"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Essential Tools: Python, Git, and Jupyter Notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "nbpresent": {
     "id": "5a4b2ca5-894e-4f77-aaf0-556fff477982"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format='retina'\n",
    "# import libraries\n",
    "import numpy as np\n",
    "import matplotlib as mp\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "from importlib import reload\n",
    "from datetime import datetime\n",
    "from IPython.display import Image\n",
    "from IPython.display import display_html\n",
    "from IPython.display import display\n",
    "from IPython.display import Math\n",
    "from IPython.display import Latex\n",
    "from IPython.display import HTML\n",
    "print('')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "![http://scikit-learn.org/stable/tutorial/machine_learning_map/](./figs/ml_map.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "1b780b57-8164-4eca-8f25-9851ed413d12"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## What you will need for this course"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "6a0e4348-5a00-4d82-9ced-3637a93cd3da"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "This course focuses on developing practical skills in working with data and providing students with a hands-on understanding of classical data analysis techniques. \n",
    "\n",
    "This will be a coding-intensive course. \n",
    "\n",
    "As discussed in Lecture 1, we are using [Python](https://www.python.org/), since it allows for fast prototyping and is supported by a great variety of scientific (and, specifically, data related) libraries."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "f9b12b68-4254-4750-8915-db3e76780913"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The materials of this course can be found under [this GitHub account](https://github.com/mcrovella/CS506-Computational-Tools-for-Data-Science). \n",
    "\n",
    "Both the lectures and the homeworks of this course are in the format of [Jupyter notebooks](http://ipython.org/notebook.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "b6077df1-2dab-46bc-bc47-890f29ab054d"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Installing Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "d17dbb84-35ea-407d-98bb-7e6ff1ee9f54"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "There are many ways to install Python.  \n",
    "\n",
    "The biggest challenge in installing Python is configuring the many different packages that can be used.\n",
    "\n",
    "For that reason, I recommend the free **Anaconda Scientific Python** distribution, which you can download from https://store.continuum.io/cshop/anaconda/. This Python distribution contains most of the packages that we will be using throughout the course. It also includes an easy-to-use but powerful package install/update system, **`conda`**.\n",
    "\n",
    "Note that we are using Python 3 (get the latest version, which should be 3.5 or later).  Be sure to download the correct version.\n",
    "\n",
    "This is what is installed on the lab machines."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "0a546e1d-5cbd-4c88-b342-5e77449a6ed4"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "a2958f76-58ff-4971-985d-9bd958f1167c"
    }
   },
   "source": [
    "One of the goals of this course is make you familiar with the workflow of code-versioning and collaboration. \n",
    "\n",
    "\n",
    "\n",
    "If you don't have it already, you should download **git** from [here](http://git-scm.com/downloads). \n",
    "\n",
    "If you don't already have one, you should also create an account on GitHub.\n",
    "\n",
    "You can find extensive documentation on how to use **git** on the [Help Pages of Github](https://help.github.com), on [Atlassian](https://www.atlassian.com/git/tutorials/setting-up-a-repository), on [GitRef](http://gitref.org/) and many other sites."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "d5840375-c301-4a2f-b5b2-025afdb97fa9"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Working with Git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "2d03ea65-575f-4adb-a5cf-c9bb5ac93ae0"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "![Alt text](./figs/git-staging-area.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "61b50409-5535-4a4f-b875-523d8cb3e486"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Configuration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "f6751685-9627-4a60-97a3-0aaa9f8bba8f"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "The first time we use *git* on a new machine, we need to configure our name and email\n",
    "\n",
    "```\n",
    "$ git config --global user.name \"Mark Crovella\"\n",
    "$ git config --global user.mail \"crovella@bu.edu\"\n",
    "```\n",
    "Use the email that you used for your GitHub account."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "216849d9-80b7-4a8a-bd1c-6491053d36ce"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Creating a Repository"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "5bcde23c-4756-48db-b89b-15957e51f93c"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "After installing Git, we can configure our first repository. First, let's create a new directory.\n",
    "\n",
    "```\n",
    "$ mkdir thoughts\n",
    "$ cd thoughts\n",
    "```\n",
    "\n",
    "Now, we can create a *git* repository in this directory.\n",
    "\n",
    "```\n",
    "$ git init\n",
    "```\n",
    "\n",
    "We can check that everything is set up correctly by asking *git* to tell us the status of our project.\n",
    "\n",
    "```\n",
    "$ git status\n",
    "On branch master\n",
    "\n",
    "Initial commit\n",
    "\n",
    "nothing to commit (create/copy files and use \"git add\" to track)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "20e30049-6480-4c72-969e-7c8d622e9263"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Now, create a file named ```science.txt```, edit it with your favorite text editor and add the following lines\n",
    "\n",
    "```\n",
    "Starting to think about data\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "e12f7ae0-554b-4ed8-9916-840b25b5d00e"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "If we check the status of our repository again, *git* tells us that there is a new file:\n",
    "```\n",
    "$ git status\n",
    "On branch master\n",
    "\n",
    "Initial commit\n",
    "\n",
    "Untracked files:\n",
    "  (use \"git add <file>...\" to include in what will be committed)\n",
    "\n",
    "        science.txt\n",
    "\n",
    "nothing added to commit but untracked files present (use \"git add\" to track)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "c1f05d9f-77cd-4e90-b976-d6e44d1e7680"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The \"untracked files\" message means that there's a file in the directory that *git* isn't keeping track of. We can tell *git* that it should do so using ```git add```:\n",
    "\n",
    "```\n",
    "$ git add science.txt\n",
    "```\n",
    "\n",
    "and then check that the file is now being tracked:\n",
    "\n",
    "```\n",
    "$ git status\n",
    "On branch master\n",
    "\n",
    "Initial commit\n",
    "\n",
    "Changes to be committed:\n",
    "  (use \"git rm --cached <file>...\" to unstage)\n",
    "\n",
    "        new file:   science.txt\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "7bbeaa7e-751d-4a56-93e6-2b6e5d768d98"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "*git* now knows that it's supposed to keep track of ```science.txt```, but it hasn't yet recorded any changes for posterity as a commit. \n",
    "\n",
    "To get it to do that, we need to run one more command:\n",
    "\n",
    "```\n",
    "$ git commit -m \"Preparing for science\"\n",
    "[master (root-commit) f516d22] Preparing for science\n",
    " 1 file changed, 1 insertion(+)\n",
    " create mode 100644 science.txt\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "f2fae4e0-2fb8-4054-8752-d2482bbdc198"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "When we run ```git commit```, *git* takes everything we have told it to save by using ```git add``` and stores a copy permanently inside the special ```.git``` directory. \n",
    "\n",
    "This permanent copy is called a **revision** and its short identifier is *f516d22*. (Your revision will have another identifier.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "64e2830e-2728-4025-9c1a-88f8709a23e9"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We use the -m flag (for \"message\") to record a comment that will help us remember later on what we did and why. \n",
    "\n",
    "If we just run ```git commit``` without the ```-m``` option, *git* will launch an editor such as ```vim``` (or whatever other editor we configured at the start) so that we can write a longer message. \n",
    "\n",
    "If you are using Windows and you are not familiar with ```vim```, try installing [GitPad](https://github.com/github/GitPad)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "2c056cc1-39af-479c-a648-3103edcddb62"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "If we run git status now:\n",
    "```\n",
    "$ git status\n",
    "On branch master\n",
    "nothing to commit, working directory clean\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "afc51b47-dbbe-4ac3-b4d3-93882b31cacd"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "it tells us everything is up to date. If we want to know what we've done recently, we can ask *git* to show us the project's history using ```git log```:\n",
    "\n",
    "```\n",
    "$ git log\n",
    "Author: Katherine Zhao <kzhao@bu.edu>\n",
    "Date:   Sun Jan 25 12:48:44 2015 -0500\n",
    "\n",
    "    Preparing for science\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "27c48abb-af66-4891-99d4-e2640db5a28a"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Changing a file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "46dd224f-3833-4452-a909-27e22c338244"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Now, suppose that we edit the file:\n",
    "\n",
    "```\n",
    "Starting to think about data\n",
    "I need to attend CS591\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "3b555fd7-1e79-4263-a4b1-7adb5eb59a8a"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Now if we run ```git status```, *git* will tell us that a file that it is tracking has been modified:\n",
    "\n",
    "```\n",
    "$ git status\n",
    "On branch master\n",
    "Changes not staged for commit:\n",
    "  (use \"git add <file>...\" to update what will be committed)\n",
    "  (use \"git checkout -- <file>...\" to discard changes in working directory)\n",
    "\n",
    "        modified:   science.txt\n",
    "\n",
    "no changes added to commit (use \"git add\" and/or \"git commit -a\")\n",
    "```\n",
    "\n",
    "The last line is the key phrase: *\"no changes added to commit\"*. \n",
    "\n",
    "We have changed this file, but we haven't told *git* we will want to save those changes (which we do with ```git add```) much less actually saved them. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "defb543e-d5d5-4ae2-ac99-b057d6648f96"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Let's double-check our work using ```git diff```, which shows us the differences between the current state of the file and the most recently saved version:\n",
    "\n",
    "```\n",
    "$ git diff\n",
    "diff --git a/science.txt b/science.txt\n",
    "index 0ac4b7b..c5b1b05 100644\n",
    "--- a/science.txt\n",
    "+++ b/science.txt\n",
    "@@ -1 +1,2 @@\n",
    " Starting to think about data\n",
    "+I need to attend CS591\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "a40271c6-d186-4179-b573-fec299bc9d8e"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "OK, we are happy with that, so let's commit our change:\n",
    "```\n",
    "$ git commit -m \"Related course\"\n",
    "On branch master\n",
    "Changes not staged for commit:\n",
    "        modified:   science.txt\n",
    "\n",
    "no changes added to commit\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "b603b598-e15d-40c7-9836-55b4e8905fd4"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "*Whoops!* *Git* won't commit the file because we didn't use ```git add``` first. Let's fix that:\n",
    "```\n",
    "$ git add science.txt\n",
    "$ git commit -m \"Related course\"\n",
    "[master 1bd7277] Related course\n",
    " 1 file changed, 1 insertion(+)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "154a475c-394b-4796-93f0-9287ceca948e"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "*Git* insists that we add files to the set we want to commit before actually committing anything because we may not want to commit everything at once. \n",
    "\n",
    "For example, suppose we're adding a few citations to our project. We might want to commit those additions, and the corresponding addition to the bibliography, but not commit the work we're doing on the analysis (which we haven't finished yet)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "9702171b-0de0-436e-b694-92e768ceb809"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "To allow for this, *git* has a special staging area where it keeps track of things that have been added to the current change set but not yet committed. ```git add``` puts things in this area, and ```git commit``` then copies them to long-term storage (as a commit):\n",
    "\n",
    "![Alt text](./figs/git-staging-area.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "0a85a94c-1749-405d-a381-3ccf78fd824c"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Recovering old versions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "9712b523-d0b0-4d82-a2c8-9a16b133032c"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We can save changes to files and see what we have changed. How can we restore older versions however? Let's suppose we accidentally overwrite the file:\n",
    "\n",
    "```\n",
    "$ cat science.txt\n",
    "Despair! Nothing works\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "309e9219-c681-447f-b151-f60402182084"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Now, ```git status``` tells us that the file has been changed, but those changes haven't been staged:\n",
    "\n",
    "```\n",
    "$ git status\n",
    "On branch master\n",
    "Changes not staged for commit:\n",
    "  (use \"git add <file>...\" to update what will be committed)\n",
    "  (use \"git checkout -- <file>...\" to discard changes in working directory)\n",
    "\n",
    "        modified:   science.txt\n",
    "\n",
    "no changes added to commit (use \"git add\" and/or \"git commit -a\")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "3305017a-ba7c-4099-911a-7d9327bb3420"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We can put things back the way they were by using ```git checkout```:\n",
    "\n",
    "```\n",
    "$ git checkout HEAD science.txt\n",
    "$ cat science.txt\n",
    "Starting to think about data\n",
    "I need to attend CS506\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "be5595f6-9ef4-497d-96dc-7a504206c702"
    }
   },
   "source": [
    "### More information"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "d41ed829-9c7d-4809-8c01-109f3d69bca7"
    }
   },
   "source": [
    "You can find more information on *git* here:\n",
    "* [Software Carpentry](http://swcarpentry.github.io/git-novice/01-backup.html)\n",
    "* [GitHub Help](https://help.github.com/)\n",
    "* [Atlassian Help](https://www.atlassian.com/git/tutorials/setting-up-a-repository)\n",
    "* [GitRef](http://gitref.org/)\n",
    "* [Git Ready](http://gitready.com/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "1b2ecb5f-c1cb-44f3-89be-04ec939e600b"
    }
   },
   "source": [
    "** IMPORTANT! **\n",
    "\n",
    "Never ```git add``` sensitive files, e.g. *passwords*, *keys*, etc., unless you are really sure you need this. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "d8eb6fe6-100b-4b06-a69b-6b21f9e892c9"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Jupyter Notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "You should get used to developing your code _incrementally._\n",
    "\n",
    "If you typically are writing files that you then run from the command line, e.g., `python code.py`, then you are not getting one of the main benefits of working in python."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "c32ab510-2293-4222-986a-189b33a0730a"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Instead you should interact with the interpreter.   \n",
    "\n",
    "A simple way to do this is to write your code in a file, but then cut and paste pieces of it into the interpreter to test."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "edf868f0-fc44-4d47-8d45-50b3a257cad6"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "To support this mode of work, use `IPython`.  It has become the standard for interactive computing in Python. \n",
    "\n",
    "There is no reason you shouldn't be using `ipython` instead of the regular python interpreter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "f8e65780-5034-4b28-b506-d5f22e28f571"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "An even better alternative is to use `Jupyter notebook`.   This puts a browser front-end on `ipython`.  \n",
    "\n",
    "These slides are actually just a Jupyter notebook.\n",
    "\n",
    "To run the Jupyter Notebook server from the command line, type ```jupyter notebook```. Your web browser will open and load the environment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "8bc5dd73-54f6-443d-97fd-249446d1c111"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "In the notebook, you can type and run code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "nbpresent": {
     "id": "c82fac99-51ab-41f4-af0d-b272bee62cdc"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hey!\n"
     ]
    }
   ],
   "source": [
    "print(\"hey!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "0c074ffe-1744-4346-8e96-adbf580b2533"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "You can use auto-complete (with the TAB key) and see the documentation (by adding  \\`?\\`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "nbpresent": {
     "id": "d20db955-eb66-4ce5-8e81-67a36b8a5b14"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['.git',\n",
       " '.ipynb_checkpoints',\n",
       " '01-Intro-to-Python.ipynb',\n",
       " '01-Intro-to-Python.pdf',\n",
       " '02A-Getting-Started.ipynb',\n",
       " '02A-Getting-Started.pdf',\n",
       " '02B-Pandas.ipynb',\n",
       " '02B-Pandas.pdf',\n",
       " '03-Probability-and-Statistics-Refresher.ipynb',\n",
       " '03-Probability-and-Statistics-Refresher.pdf',\n",
       " '04-Linear-Algebra-Refresher.ipynb',\n",
       " '04-Linear-Algebra-Refresher.pdf',\n",
       " '05-Distances-Timeseries.ipynb',\n",
       " '05-Distances-Timeseries.pdf',\n",
       " '06A-Clustering-Overview.ipynb',\n",
       " '06A-Clustering-Overview.pdf',\n",
       " '06B-Kmeans.pdf',\n",
       " '07-Clustering-II.ipynb',\n",
       " '07-Clustering-II.pdf',\n",
       " '08A-Clustering-III-Hierarchical.pdf',\n",
       " '08B-Clustering-III.ipynb',\n",
       " '08B-Clustering-III.pdf',\n",
       " '08C-Hierarchical-Example.pdf',\n",
       " '09-Clustering-IV-GMM-EM.ipynb',\n",
       " '09-Clustering-IV-GMM-EM.pdf',\n",
       " '10-Low-Rank-and-SVD.ipynb',\n",
       " '10-Low-Rank-and-SVD.pdf',\n",
       " '11-Dimensionality-Reduction-SVD-II.ipynb',\n",
       " '11-Dimensionality-Reduction-SVD-II.pdf',\n",
       " '12-Anomaly-Detection-SVD-III.ipynb',\n",
       " '12-Anomaly-Detection-SVD-III.pdf',\n",
       " '13-Web-Scraping.pdf',\n",
       " '14-Classification-Decision-Trees.pdf',\n",
       " '15A-Classification-II-kNN.pdf',\n",
       " '15B-Classification-II-Demo.ipynb',\n",
       " '15B-Classification-II-Demo.pdf',\n",
       " '16A-Classification-III-NB-SVM.pdf',\n",
       " '16B-Classification-III-Random-Projection.ipynb',\n",
       " '16C-Classification-III-SVM-Demo.ipynb',\n",
       " '16C-Classification-III-SVM-Demo.pdf',\n",
       " '17-Regression-I-Linear.ipynb',\n",
       " '17-Regression-I-Linear.pdf',\n",
       " '18-Regression-II-Logistic.ipynb',\n",
       " '18-Regression-II-Logistic.pdf',\n",
       " '19-Regression-III-More-Linear.ipynb',\n",
       " '19-Regression-III-More-Linear.pdf',\n",
       " '20-Recommender-Systems-Edited.pdf',\n",
       " '20-Recommender-Systems.pdf',\n",
       " '21A-MapReduce.pdf',\n",
       " '21B-MapReduce-Demo.ipynb',\n",
       " '22A-Networks-I-Intro-Edited.pdf',\n",
       " '22A-Networks-I-Intro.pdf',\n",
       " '22B-Networks-I-Intro-Demo.ipynb',\n",
       " '22B-Networks-I-Intro-Demo.pdf',\n",
       " '23-Networks-II-Centrality.ipynb',\n",
       " '23-Networks-II-Centrality.pdf',\n",
       " '24-Networks-III-Clustering.ipynb',\n",
       " '24-Networks-III-Clustering.pdf',\n",
       " '25-Networks-MapReduce.pdf',\n",
       " '__pycache__',\n",
       " 'auto',\n",
       " 'CS505FinalProject-Fall2016.pdf',\n",
       " 'data',\n",
       " 'example.txt',\n",
       " 'figs',\n",
       " 'img.jpg',\n",
       " 'laUtilities.py',\n",
       " 'Makefile',\n",
       " 'Old',\n",
       " 'postamble.tex',\n",
       " 'preamble.tex',\n",
       " 'README.md',\n",
       " 'slideUtilities.py',\n",
       " 'stripheaders.py',\n",
       " 'stripHiddenCode.py',\n",
       " 'syllabus.aux',\n",
       " 'syllabus.dvi',\n",
       " 'syllabus.log',\n",
       " 'syllabus.pdf',\n",
       " 'syllabus.pdf~',\n",
       " 'syllabus.ps',\n",
       " 'syllabus.tex',\n",
       " 'yahoo_data.csv']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "os.listdir()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "1e6f5bd8-1940-4523-a37b-90996bb2dba1"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The errors are nicely formatted:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "nbpresent": {
     "id": "ade68081-a94f-43e0-8e6d-284eb881307c"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "#1/0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "da453106-13e3-4ed3-9b4e-9d713f5427dd"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Essential Shortcuts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "c59bb8a8-272a-491f-8f83-0552e52a7aa8"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "* *Esc / Enter*: Switch between edit and command mode\n",
    "* Execute cells\n",
    "    * *Shift-Enter*: Run and move to the next cell\n",
    "    * *Alt-Enter*: Run and make new cell\n",
    "    * *Ctrl-Enter*: Run in place\n",
    "* *a / b* : Insert cell below / above\n",
    "* *d*: Delete cell"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "217b6277-35d2-4bfd-b52d-cf6ad0b7fd4a"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Working from the Undergraduate lab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "f94a408a-51de-43fa-b2c1-c1b15c9d81aa"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Anaconda has been installed in the Linux machines of the Undergraduate lab as well. If you want to work from there, you need to follow the next steps:\n",
    "\n",
    "1. Add the ```conda``` executable to your PATH\n",
    "```\n",
    "$ export PATH=/usr/local/anaconda/bin:$PATH\n",
    "```\n",
    "\n",
    "2. Create a new environment (only do this once)\n",
    "``` \n",
    "$ conda create -p ~/envs/test numpy scipy networkx pandas  scikit-learn matplotlib beautiful-soup ipython-notebook=2.2\n",
    "```\n",
    "You can change its name to something other than ```test```.\n",
    "\n",
    "3. Activate the environment\n",
    "```\n",
    "$ source activate ~/envs/test\n",
    "```\n",
    "\n",
    "4. Run IPython or IPython notebook\n",
    "```\n",
    "$ jupyter notebook --ip=127.0.0.1\n",
    "```\n",
    "\n",
    "5. Deactivate the environment\n",
    "```\n",
    "$ source deactivate\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "63469f5c-77a5-491d-a3bb-6bcc28aca70d"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## GitHub"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "0b741034-531d-4538-bc1f-b573a4eab34d"
    }
   },
   "source": [
    "Systems like *git* allow us to move work between any two repositories.\n",
    "\n",
    "In practice, though, it's easiest to use one copy as a central hub, and to keep it on the web rather than on someone's laptop. Most programmers use hosting services like [GitHub](https://github.com/) or [BitBucket](https://bitbucket.org/) to hold those master copies. \n",
    "\n",
    "For the purpose of our course, we will be using [GitHub](https://github.com/) to host the course material. \n",
    "\n",
    "You will also submit your homeworks through this platform. \n",
    "\n",
    "Next, we will cover how you can fork and clone the course's repository and how to submit your solutions to the homework. \n",
    "\n",
    "For more information on how to create your own repository on GitHub and upload code to it, please see the tutorial by [Software Carpentry](http://swcarpentry.github.io/git-novice/02-collab.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "04460419-9b64-4fd9-a1e1-3dd29c00105a"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Course repositories"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "4b173db4-8824-4065-ba0c-ab406fb8f544"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "The material of the course is hosted on GitHub, under [this account](https://github.com/mcrovella/CS506-Computational-Tools-for-Data-Science)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "3f41c6d6-a535-4a07-b92b-3d34977fce6a"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Clone the lecture repository"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "559240af-1ae4-4f7b-b47b-4967bca0fe30"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "In order to download a copy of the lectures and run them locally on your computer, you need to clone the lecture repository. To do that:\n",
    "\n",
    "1. Create a new folder for the course.\n",
    "```\n",
    "$ mkdir cs506\n",
    "$ cd cs506\n",
    "```\n",
    "2. Copy the clone url from the [repository's website](https://github.com/mcrovella/CS506-Computational-Tools-for-Data-Science).\n",
    "3. Clone the repository from *git*.\n",
    "```\n",
    "$ git clone https://github.com/mcrovella/CS506-Computational-Tools-for-Data-Science.git\n",
    "```\n",
    "\n",
    "You should now have a directory with the course material.\n",
    "\n",
    "To update the repository and download the **new material**, type\n",
    "```\n",
    "$ git pull\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "6c03a8ae-289c-41f0-9469-6002b0c1fc9c"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Create a private homework repository"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "9725ece4-64ba-4afb-be06-fba207bc94e8"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Each assignment will have its own private repository, on GitHub shared only by you and the instructors.  \n",
    "\n",
    "The last commit time on the repo will be used as the submission time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "99393653-fcfe-4c2b-84d5-71c8184701f1"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## Practice"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "4d8e3733-fea4-495d-81e2-f060405e234d"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Now, practice what we have seen today by solving and submitting Homework 0."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  },
  "nbpresent": {
   "slides": {
    "5f3c7f98-1925-4a75-bba3-ce201e3cd531": {
     "id": "5f3c7f98-1925-4a75-bba3-ce201e3cd531",
     "prev": null,
     "regions": {
      "74a87d20-d055-4a94-b37b-d140a6e83ea5": {
       "attrs": {
        "height": 1,
        "width": 1,
        "x": 0,
        "y": 0
       },
       "content": {
        "cell": "eefcedb2-d95e-406f-adf0-a889c096010c",
        "part": "source"
       },
       "id": "74a87d20-d055-4a94-b37b-d140a6e83ea5"
      }
     }
    },
    "fd5c2b76-6eae-429d-91b6-b9669434839b": {
     "id": "fd5c2b76-6eae-429d-91b6-b9669434839b",
     "prev": "5f3c7f98-1925-4a75-bba3-ce201e3cd531",
     "regions": {
      "2ce5e167-6174-4a61-885f-66e8c39554c3": {
       "attrs": {
        "height": 1,
        "width": 1,
        "x": 0,
        "y": 0
       },
       "content": {
        "cell": "1b780b57-8164-4eca-8f25-9851ed413d12",
        "part": "source"
       },
       "id": "2ce5e167-6174-4a61-885f-66e8c39554c3"
      }
     }
    }
   },
   "themes": {}
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}