Spike sorting pipeline.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Spike Sorting Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook runs you through the process of semi-automatic spike sorting pipeline. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1) Copy your data from HDD to SSD \n",
    "\n",
    "First things first! Before any data analysis, we need to <b>copy the data from the hard drive where the raw data is stored to the 1 TB SSD on this computer that is designated as the workspace</b>. This is asked for two reasons: \n",
    "\n",
    "<ol type=\"1\">\n",
    "<li>There is a remarkable difference between the speed of reading data from the HDD vs. SSD</li>\n",
    "<li>There are some by-products of the data analysis that should be deleted at the end of the analysis. If these files are stored in the HDD, Dropbox tries to synchonize all those files as well. And also some times these files are forgotten to be deleted and end up taking precious space on the HDD where more novel, raw data could have been stored.</li>\n",
    "</ol>\n",
    "\n",
    "<b> IF </b> you have completed this step, please continue with the following steps."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2) Generate parameter dictionary for all recording sessions\n",
    "\n",
    "Next, we will need to generate pickle files named <i> paramsDict.p </i> for each recording session in your experiment. These pickle files contain crucial parameters related to data acquisition and your preferences on the details of how the data should be analyzed. For this procedure, please <b> run the block below </b> to launch the Jupyter notebook (Python 3) for generating the parameter dictionary. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "source activate klusta #Activating the klustakwik virtual environment\n",
    "jupyter notebook './Generate_dict_for_experiment.ipynb'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  2.5) Restart kernel\n",
    "\n",
    "Since no bash or python command to this date exists for properly closing a jupyter notebook, from the <i> Kernel </i> tab above, select <i> Restart & Clear Output </i> to restart the kernel. That will not be an issue since the <i> paramsDict.p </i> files are already generated and saved."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3) Run the main analysis function on the data \n",
    "\n",
    "We are ready to perform the analysis on the data. Please first <b> specify the path (with / at the end) </b> to the folder that contains the folders for all recprding sessions (i.e. the folder right above the folders of recording sessions in hierarchy) and then <b> run the following line of code </b> to run the script <i> analyze_all_recording_sessions.py </i> which will run the <i> main </i> function in the <i> main_tetrode.py </i>. You can check the scripts for the details of the steps running on the background. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Traceback (most recent call last):\n",
      "  File \"utils/analyze_all_recording_sessions.py\", line 14, in <module>\n",
      "    from main import main \n",
      "  File \"/home/baran/Desktop/git-repos/yaniklab-ephys/utils/main.py\", line 15, in <module>\n",
      "    from spikeSortingUtils.klusta_preprocessing_utils import *\n",
      "ModuleNotFoundError: No module named 'spikeSortingUtils'\n"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "#Specify the path here!\n",
    "#PATHEXP=\"/path/to/the/folder/containing/the/folders/for/recording/sessions/\" \n",
    "PATHEXP='/home/baran/Desktop/2017_03_25_Barrel_cortex/'\n",
    "echo $PATHEXP|python utils/analyze_all_recording_sessions.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4) Initial manual clustering\n",
    "\n",
    "As we have run the Klustakwik, now we are ready to generate figures and  extract results for this experiment. It is time to take a look at the results of the initial clustering to <b> eliminate obvious noise clusters</b>. At this stage, <b>please do not attempt to make conclusive judgements</b> about the cluster identities since it will happen at a later step that is also guided by our custom GUI.\n",
    "\n",
    "The cell below activates the klustaviewa environment and also provides a list of recording session folders inside the experiment folder. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "source activate klustaviewa\n",
    "ls $PATHEXP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please run the klustaviewa command on the tetrodes/shanks of every recording session that you would like to perform spike sorting on, from the experiment. <b>Run the klustaviewa command on the .kwik file inside the folder for the shank or the tetrode</b>, as shown in the example below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "klustaviewa '/home/baran/Desktop/2017_03_25_Barrel_cortex/spont_170325_162858/probe_0_shank_2/probe_0_shank_2.kwik'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After finishing the initial manual clustering process for a kwik file, <b>save the results and exit klustaviewa</b>. Then, <b>run the cell above again for the next .kwik file to be analyzed</b>. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "With the remaining clusters, we can extract the spike times and the waveforms from the .clu files first. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "echo $PATHEXP|python extract_spikeinfo_from_all.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5) Main manual clustering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next step is to run the Klustaviewa and the custom GUI together on each recording session to make a conclusive judgment about the clusters. <b>Use the GUI as complementary information and make the final mergings and cluster assignments on Klustaviewa</b>. Please check the wiki for further information on Spike Sorting. At the end of the process, save your results on Klustaviewa to obtain the final .clu files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "PATHSESSION = '/home/baran/Desktop/2017_03_25_Barrel_cortex/spont_170325_162858/probe_0_shank_2/probe_0_shank_2.kwik'\n",
    "klustaviewa $PATHSESSION\n",
    "echo $PATHSESSION|python gui.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6) Saving the spike times and waveforms for the final clusters\n",
    "\n",
    "As the final clusters have been determined, it is time to save the spike times and waveforms for these final clusters. Run the following block to do that for all recording sessions. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "echo $PATHEXP|python extract_spikeinfo_from_all.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7) Quality measurement of the sorted units\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Done! \n",
    "\n",
    "You can check the <i> Analyzed </i> folder to see the plots and the excel sheets containing the evoked LFP data and <b> move the files and folders </b> that you deem necessary into the <i> Analyzed </i> folder inside the <i> Electrophysiology </i> Dropbox folder. Please do not forget to rename the folder with the date and the name of the experiment when adding to the <i> Analyzed </i> folder. At the end, please <b> delete the data and the intermediate files from the SSD. </b> \n",
    "\n",
    "Notebook written by Baran Yasar in 04/2017. Please contact him in person or via e-mail at yasar@biomed.ee.ethz.ch in case of any questions."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}