02_how-medgan-binary-works.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"top\"></a>\n",
    "<br/>\n",
    "# Understanding how `medGAN` works on the MIMIC-III dataset of shape (46 520, 1 071) with binary values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Author: [Sylvain Combettes](https://github.com/sylvaincom). <br/>\n",
    "Last update: Sep 11, 2019. Creation: Aug 12, 2019. <br/>\n",
    "My own `medGAN` repository: [medgan-tips](https://github.com/sylvaincom/medgan-tips) (based on Edward Choi's work). <br/>\n",
    "Edward Choi's original repository: [medgan](https://github.com/mp2893/medgan).\n",
    "\n",
    "The final goal of my project is to use `medGAN` on my own dataset (patient data). Hence, I first need to understand how the `medGAN` program works. In this notebook, I provide code cells and explanations to help better understand and run `medGAN`. Because there are some confidentiality issues with the MIMIC-III dataset, I cleared the output of the cells.\n",
    "\n",
    "Before reading this notebook, make sure that you have read my [medGAN repository](https://github.com/sylvaincom/medgan-tips)'s table of contents.\n",
    "\n",
    "We will use the MIMIC-III dataset and process it so that we only have binary values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Table of contents\n",
    "\n",
    "- [1) Loading the MIMIC-III dataset](#load-mimic)\n",
    "- [2) Using `process_mimic.py` and `medgan.py` to generate the fake realistic data](#run)\n",
    "- [3) How can one interpret the output of `medgan.py`?](#gen-samples)\n",
    "- [4) How can one interpret the output of `process_mimic.py`?](#input)\n",
    "- [5) Comparing the fictitious generated samples to the real-life original ones](#comparison)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import os\n",
    "import pickle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "<a name=\"load-mimic\"></a>\n",
    "# 1) Loading the MIMIC-III dataset\n",
    "\n",
    "## 1.1) `ADMISSIONS.csv`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_adm = pd.read_csv(\"ADMISSIONS.csv\")\n",
    "print(df_adm.shape)\n",
    "df_adm.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do we have a lot of missing values?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def observe_missing_values(df):\n",
    "    # This function does not return any values.\n",
    "    \n",
    "    dico = {}\n",
    "    c_index = [] # index of columns of features with missing values\n",
    "    n,p = df.shape\n",
    "    features = list(df) # list of features\n",
    "    \n",
    "    # We compute the percentage of missing values for each column:\n",
    "    for f in features:\n",
    "        percentage = round(sum(df[f].isna())*100/n, 2) # percentage of missing values\n",
    "        if percentage>0:\n",
    "            dico[f] = [percentage]\n",
    "            c_index.append(df.columns.get_loc(f))\n",
    "    \n",
    "    if c_index == []:\n",
    "        return 'There are no missing values!'\n",
    "    \n",
    "    # We construct the DataFrame ordered by Approx. missing values (%):\n",
    "    df_mv = pd.DataFrame(data=dico) # mv for missing values\n",
    "    idx_rename = {df_mv.index.tolist()[0]:'Approx. missing values (%)'}\n",
    "    df_mv = df_mv.rename(index=idx_rename)\n",
    "    \n",
    "    # We print the features with a decreasing Approx. missing values (%) value:\n",
    "    n_f = min(p,10) # number of features we choose to print\n",
    "    df_mv_ordered = df_mv.sort_values(by=['Approx. missing values (%)'], axis=1, ascending=False)\n",
    "    print('The', n_f, 'features with the most missing values are:')\n",
    "    print(df_mv_ordered.iloc[0].head(n_f))\n",
    "    \n",
    "    # We plot the Approx. missing values (%) given the index of the feature:\n",
    "    plt.plot(c_index, df_mv.iloc[0], 'o')\n",
    "    plt.xlabel('Index of feature')\n",
    "    plt.ylabel('Approx. missing values (%)')\n",
    "    plt.title('Observing missing values')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "observe_missing_values(df_adm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2) `DIAGNOSES_ICD.csv`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_ICD = pd.read_csv('DIAGNOSES_ICD.csv')\n",
    "print(df_ICD.shape)\n",
    "df_ICD.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do we have a lot of missing values?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "observe_missing_values(df_ICD)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We check if our dataset is balanced. Does one `ICD9_CODE` appear distinctly more than others in proportion?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_ICD['ICD9_CODE'].value_counts(normalize=True).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# 2) Using `process_mimic.py` and `medgan.py` to generate the fake realistic data <a name=\"run\"></a>\n",
    "\n",
    "This step is detailed in my tutorial [A few additional tips on how to run Edward Choi's medGAN](https://github.com/sylvaincom/medgan/blob/master/tips-for-medgan.md).\n",
    "\n",
    "In short, in the Anaconda prompt, we run:\n",
    "```\n",
    "cd C:\\Users\\<username>\\Documents\\mimic_binary\n",
    "python process_mimic.py ADMISSIONS.csv DIAGNOSES_ICD.csv training-data \"binary\"\n",
    "mkdir generated\n",
    "python medgan.py training-data.matrix ./generated/samples --data_type=\"binary\"\n",
    "python medgan.py training-data.matrix gen-samples --model_file=./generated/samples-999 --generate_data=True --data_type=\"binary\"\n",
    "```\n",
    "Some default values are `n_epoch=1000`, `n_pretrain_epoch=100` and `batch_size=1000`. We choose `nSamples=10000` line 405 in `medgan.py`.\n",
    "The computing took more than 5 hours on my laptop."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# 3) How can one interpret the output of `medgan.py`? <a name=\"gen-samples\"></a>\n",
    "\n",
    "We load the `gen-samples.npy` file which is `medgan.py`'s output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fict = np.load('gen-samples.npy') # fictitious generated dataset\n",
    "df_fict = pd.DataFrame(fict)\n",
    "print(df_fict.shape)\n",
    "df_fict.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do the output of `medgan.py` have missing values?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "observe_missing_values(df_fict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output of `medgan.py` has no missing values!\n",
    "\n",
    "Some questions about this data frame:\n",
    "* What do the columns correspond to? They are not the ones of `ADMISSIONS.csv` nor `DIAGNOSIS_ICD.csv`.\n",
    "* What do the rows correspond to?\n",
    "* Why are the values not binary?\n",
    "\n",
    "We can find some answers in an issue opened in Edward Choi's GitHub: [How to interpret the samples?](https://github.com/mp2893/medgan/issues/3). In order to understand the output `gen-samples.npy` of `medgan.py`, we are going to go back to the input of `medgan.py` which is the output of `process_mimic.py`: `training-data.matrix`.\n",
    "\n",
    "Actually, in `gen-samples.npy`, as in the `training-data.matrix` file, each row corresponds to a single patient and each column corresponds to a specific ICD9 diagnosis code. We can use the `training-data.types` file created by `process_mimic.py` to map each column to a specific ICD9 diagnosis code. Read the beginning part of the source code of `process_mimic.py` for more information about these files:\n",
    "```python\n",
    "# Output files\n",
    "# <output file>.pids: cPickled Python list of unique Patient IDs. Used for intermediate processing\n",
    "# <output file>.matrix: Numpy float32 matrix. Each row corresponds to a patient. Each column corresponds to a ICD9 diagnosis code.\n",
    "# <output file>.types: cPickled Python dictionary that maps string diagnosis codes to integer diagnosis codes.\n",
    "```\n",
    "\n",
    "What is ICD-9? See [ICD-9](https://en.wikipedia.org/wiki/International_Statistical_Classification_of_Diseases_and_Related_Health_Problems#ICD-9) and [List of ICD-9 codes](https://en.wikipedia.org/wiki/List_of_ICD-9_codes).\n",
    "\n",
    "We need to round the values of the generated dataset ourselves:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_fict = df_fict.round(0)\n",
    "df_fict.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Indeed, in line 405 of `medgan.py`, we have chosen `nSamples=10000`. It is important to note that once we have learned the estimated density of the real-life original samples, we can generate as many fake realistic samples as we want."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# 4) How can one interpret the output of `process_mimic.py`? <a name=\"input\"></a>\n",
    "\n",
    "## 4.1) Understanding `training-data.types`\n",
    "\n",
    "_cPickled Python dictionary that maps string diagnosis codes to integer diagnosis codes._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "map_dict = pickle.load(open('training-data.types', 'rb'))\n",
    "print(type(map_dict))\n",
    "print('An excerpt is:', dict(list(map_dict.items())[0:5]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thus, as its name suggests, `process_mimic.py` is really dependent on the MIMIC-III dataset. We will not use `process_mimic.py` on our own dataset and only run `medgan.py`. Out of `process_mimic.py`, we only need to understand how the output file `training-data.matrix` is constructed.\n",
    "\n",
    "## 4.2) Understanding `training-data.pids`\n",
    "\n",
    "_cPickled Python list of unique Patient IDs. Used for intermediate processing_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "id_list = pickle.load(open('training-data.pids', 'rb'))\n",
    "print(type(id_list))\n",
    "print('An excerpt is:', id_list[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This list is not useful for our own dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.3) Understanding `training-data.matrix`\n",
    "\n",
    "_Numpy float32 matrix. Each row corresponds to a patient. Each column corresponds to a ICD9 diagnosis code._\n",
    "\n",
    "`training-data.matrix` is an output of `process_mimic.py` and the input of `medgan.py`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "real_data = pickle.load(open('training-data.matrix', 'rb')) # real-life original dataset\n",
    "print(type(real_data))\n",
    "real_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_real = pd.DataFrame(real_data)\n",
    "print(df_real.shape)\n",
    "df_real.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we chose, the real-life data is binary. \n",
    "\n",
    "We can note that the input and the [output](#gen-samples) of `medgan.py` have the same number of columns and that the values are of the same type (binary). Thus, `gen-samples.npy` is a fictitious realistic generated dataset that is trying to estimate the data in the `training-data.matrix` file.\n",
    "\n",
    "Does the input of `medgan.py` have missing values?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "observe_missing_values(df_real)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The input of `medgan.py` has no missing values!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "<a name=\"comparison\"></a>\n",
    "# 5) Comparing the fictitious generated samples to the real-life original ones\n",
    "\n",
    "We wish to measure the accuracy of the fictitious generated dataset called `df_fict` considering the real-life original one called `df_real`. Is our fictitious generated dataset realistic?\n",
    "\n",
    "Here is a recap of our parameters for `medGAN`:\n",
    "\n",
    "| dataset | number of samples | number of features |\n",
    "|---|---|---|\n",
    "|`df_real` | 46 520 | 1 071 |\n",
    "|`df_fict` | 10 000 | 1 071 |\n",
    "\n",
    "| `n_epoch` | `n_pretrain_epoch` | `batch_size` | `nSamples` |\n",
    "|---|---|---|---|\n",
    "| 1 000 | 100 | 1 000 | 10 000 |\n",
    "\n",
    "As in Choi's [paper](https://arxiv.org/abs/1703.06490), we use dimension-wise probability as a measure of accuracy. Indeed, the variables are binary. Given that our data is binary, for each feature (dimension), we claim that 1 corresponds to success and 0 to failure. Hence the proportion of 1s obtained is the Bernoulli success probability $p$.\n",
    "\n",
    "## 5.1) Probability distribution of the real-life data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_real = df_real\n",
    "n_real, p_real = df_real.shape\n",
    "print(n_real, p_real)\n",
    "\n",
    "proba_real = [sum(df_real[f])/n_real for f in list(df_real)]\n",
    "\n",
    "plt.plot(proba_real, 'o')\n",
    "plt.xlabel('Index of feature')\n",
    "plt.ylabel('Bernoulli probability success')\n",
    "plt.title('For the real-life dataset')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.2) Probability distribution of the fictitious generated data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_fict, p_fict = df_fict.shape\n",
    "print(n_fict, p_fict)\n",
    "\n",
    "proba_fict = [sum(df_fict[f])/n_fict for f in list(df_fict)]\n",
    "\n",
    "plt.plot(proba_fict, 'o')\n",
    "plt.xlabel('Index of feature')\n",
    "plt.ylabel('Bernoulli probability success')\n",
    "plt.title('For the fictitious generated dataset')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.3) Comparison: dimension-wise probability"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xaxis = proba_real\n",
    "yaxis = proba_fict\n",
    "\n",
    "start = min(np.min(xaxis), np.min(yaxis))\n",
    "stop = max(np.max(xaxis), np.max(yaxis))\n",
    "p = len(xaxis)\n",
    "X = np.linspace(start, stop, num=p+1)\n",
    "\n",
    "plt.plot(xaxis, yaxis, 'ok', X, X, '-g');\n",
    "\n",
    "plt.legend(['Bernoulli success probability', 'ideal Bernoulli success probability'])\n",
    "plt.title('Dimension-wise probability performance of medGAN')\n",
    "plt.xlabel('For the real-life dataset')\n",
    "plt.ylabel('For the fictitious generated dataset')\n",
    "plt.savefig('accuracy_mimic_binary.png', dpi=120) # to save the image in high resolution\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The values of the $x$-axis and the $y$-axis are ordered: we successively compare the Bernoulli success probability for both datasets (`df_real` and `df_fict`) for a given feature. For information, we have 1 071 features thus 1 071 scatter points.\n",
    "\n",
    "The diagonal green line indicates the ideal performance where the real and the (fake) realistic generated data show identical proportions of 1s. Based on the graph, as the dots are close to the diagonal green line, we can say that `medGAN` has a really good performance. I recall that we have a total of 1 071 points so it does not matter if we have a few points that are far from the diagonal green line.\n",
    "\n",
    "> *Experimental result* - **The synthesis of binary values using `medGAN` works.** <br/>\n",
    "We have observed that `medGAN` can generate fictitious samples with binary values that are realistic.\n",
    "\n",
    "> *Further work* - **We could quantify the accuracy of the generated dataset.** <br/>\n",
    "Rather than observing the accuracy of `medGAN` on a graph, we could quantify the accuracy by computing the total error: the sum (on all features) of the squares of the Bernoulli success probability difference / error. Graphically, an individual error (for a given feature) is the distance from the dot to the diagonal green line. We can normalize this total error by dividing it by the total number of features. This normalized squared error is actually the [MSE](https://en.wikipedia.org/wiki/Mean_squared_error) (mean squared error).\n",
    "We do not compute the MSE here because we have nothing to compare its value to."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "Back to [top](#top)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}