Copy_of_FMML_CLF1_Lab3_2022.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/nagababumo/iiitfmmlassignments/blob/main/Copy_of_FMML_CLF1_Lab3_2022.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kAKrrEmgLZ_J"
      },
      "source": [
        "# **FOUNDATIONS OF MODERN MACHINE LEARNING, IIIT Hyderabad**\n",
        "### MODULE: CLASSIFICATION-1\n",
        "### LAB-3 : Using KNN for Text Classification\n",
        "#### Module Coordinator: Tanvi Kamble\n",
        "\n",
        "\n",
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Section 1: Understanding NLP tools** "
      ],
      "metadata": {
        "id": "_jeRAsurrdLY"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "In this lab we will be using KNN on a real world NLP application i.e. is text classification. But first look at some NLP techniques for text classification and tools that we use when we want to use python for NLP."
      ],
      "metadata": {
        "id": "iB9N2m-HrgWY"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WMJhTF89MmT_"
      },
      "source": [
        "## Section 1.2: Data Cleaning and Preprocessing step"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nRXVjJHk3urf"
      },
      "source": [
        "Raw text must be processed and converted into a form so that it is suitable to use with various machine-learning algorithms.  \n",
        "In case of text, there are lots of things that need to be taken into account.  \n",
        "\n",
        "\n",
        "1.   Removing numbers from the text\n",
        "2.   Handling capitalization and punctuation.\n",
        "3.   Stemming and Lemmatizing text.  \n",
        "\n",
        "And most importantly, one can't just use words or images directly in algorithms; they need to be converted into vectors- a form that algorithms can understand.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **NLTK** \n",
        "NLTK (or Natural Language Tool Kit) is a commonly used library for processing text. We will use this tool in this lab. Lets first install it. \n"
      ],
      "metadata": {
        "id": "IYq4np0xqUBr"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wFOZHWV3_ABt",
        "outputId": "2413ee75-9cbf-482a-d489-e7aa9e1c73a0"
      },
      "source": [
        "import nltk\n",
        "nltk.download('stopwords')\n",
        "nltk.download('punkt')\n",
        "nltk.download('averaged_perceptron_tagger')\n",
        "nltk.download('wordnet')\n",
        "nltk.download('omw-1.4')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/stopwords.zip.\n",
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n",
            "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
            "[nltk_data]     /root/nltk_data...\n",
            "[nltk_data]   Unzipping taggers/averaged_perceptron_tagger.zip.\n",
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data] Downloading package omw-1.4 to /root/nltk_data...\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 1
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Gpgq3SQK5IOr"
      },
      "source": [
        "import re\n",
        "import numpy\n",
        "from nltk.corpus import stopwords\n",
        "from nltk.stem import WordNetLemmatizer\n",
        "from nltk.stem import SnowballStemmer\n",
        "from nltk.tokenize import word_tokenize\n",
        "from nltk import pos_tag\n",
        "from nltk.corpus import wordnet\n",
        "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\n",
        "from bs4 import BeautifulSoup\n",
        "\n",
        "def cleanText(text, lemmatize, stemmer):\n",
        "    \"\"\"Method for cleaning text from train and test data. Removes numbers, punctuation, and capitalization. Stems or lemmatizes text.\"\"\"\n",
        "\n",
        "    if isinstance(text, float):\n",
        "        text = str(text)\n",
        "    if isinstance(text, numpy.int64):\n",
        "        text = str(text)\n",
        "    try:\n",
        "        text = text.decode()\n",
        "    except AttributeError:\n",
        "        pass\n",
        "\n",
        "    soup = BeautifulSoup(text, \"lxml\")\n",
        "    text = soup.get_text()\n",
        "    text = re.sub(r\"[^A-Za-z]\", \" \", text)\n",
        "    text = text.lower()\n",
        "\n",
        "\n",
        "    if lemmatize:\n",
        "        wordnet_lemmatizer = WordNetLemmatizer()\n",
        "\n",
        "        def get_tag(tag):\n",
        "            if tag.startswith('J'):\n",
        "                return wordnet.ADJ\n",
        "            elif tag.startswith('V'):\n",
        "                return wordnet.VERB\n",
        "            elif tag.startswith('N'):\n",
        "                return wordnet.NOUN\n",
        "            elif tag.startswith('R'):\n",
        "                return wordnet.ADV\n",
        "            else:\n",
        "                return ''\n",
        "\n",
        "        text_result = []\n",
        "        tokens = word_tokenize(text)  # Generate list of tokens\n",
        "        tagged = pos_tag(tokens)\n",
        "        for t in tagged:\n",
        "            try:\n",
        "                text_result.append(wordnet_lemmatizer.lemmatize(t[0], get_tag(t[1][:2])))\n",
        "            except:\n",
        "                text_result.append(wordnet_lemmatizer.lemmatize(t[0]))\n",
        "        return text_result\n",
        "\n",
        "    if stemmer:\n",
        "        text_result = []\n",
        "        tokens = word_tokenize(text)\n",
        "        snowball_stemmer = SnowballStemmer('english')\n",
        "        for t in tokens:\n",
        "            text_result.append(snowball_stemmer.stem(t))\n",
        "        return text_result"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "sample_text = \"Troubling\"\n",
        "sample_text_result = cleanText(sample_text, lemmatize=False, stemmer=True)\n",
        "sample_text_result = \" \".join(str(x) for x in sample_text_result)\n",
        "print(sample_text)\n",
        "print(sample_text_result)\n",
        "sample_text_result = cleanText(sample_text, lemmatize=True, stemmer=False)\n",
        "sample_text_result = \" \".join(str(x) for x in sample_text_result)\n",
        "print(sample_text_result)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZuBxfzbuzrs1",
        "outputId": "d23f655f-14da-444e-c4f7-959ab5cf2390"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Troubling\n",
            "troubl\n",
            "trouble\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QcqQi34UoPvq"
      },
      "source": [
        "## Section 1.2: BAG OF WORDS \n",
        "\n",
        "A bag-of-words model, or BoW for short, is a way of extracting features from text for use in modeling, such as with machine learning algorithms.\n",
        "\n",
        "The approach is very simple and flexible, and can be used in many ways for extracting features from documents.\n",
        "\n",
        "A bag-of-words is a representation of text that describes the occurrence of words within a document. \n",
        "It is called a “bag” of words, because any information about the order or structure of words in the document is discarded. The model is only concerned with whether known words occur in the document, not where in the document."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3YYSpQzIM05l"
      },
      "source": [
        "# Functions to convert document(s) to a list of words, with the option of removing stopwords. Returns document-term matrix.\n",
        "\n",
        "def createBagOfWords(train, test, remove_stopwords, lemmatize, stemmer):\n",
        "    if remove_stopwords:\n",
        "        vectorizer = CountVectorizer(analyzer='word', input='content', stop_words=stopwords.words('english'))\n",
        "    else:\n",
        "        vectorizer = CountVectorizer(analyzer='word', input='content')\n",
        "\n",
        "    clean_train = []\n",
        "    for paragraph in train:\n",
        "        paragraph_result = cleanText(paragraph, lemmatize, stemmer)\n",
        "        paragraph = \" \".join(str(x) for x in paragraph_result)\n",
        "        clean_train.append(paragraph)\n",
        "\n",
        "    clean_test = []\n",
        "    for paragraph in test:\n",
        "        paragraph_result = cleanText(paragraph, lemmatize, stemmer)\n",
        "        paragraph = \" \".join(str(x) for x in paragraph_result)\n",
        "        clean_test.append(paragraph)\n",
        "\n",
        "    bag_of_words_train = vectorizer.fit_transform(clean_train).toarray()\n",
        "    bag_of_words_test = vectorizer.transform(clean_test).toarray()\n",
        "    return bag_of_words_train, bag_of_words_test\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-v0iCUBqoX82"
      },
      "source": [
        "## Section 1.3: TF-IDF \n",
        "TF-IDF technique is used to find meaning of sentences consisting of words and cancels out the incapabilities of Bag of Words technique which is good for text classification or for helping a machine read words in numbers.\n",
        "\n",
        "The number of times a term occurs in a document is called its Term frequency (TF).\n",
        "\n",
        " Document frequency is the number of documents in which the word is present.  Inverse DF (IDF) is the inverse of the document frequency which measures the informativeness of term *t*. \n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L6_-DDMQoXEX"
      },
      "source": [
        "def createTFIDF(train, test, remove_stopwords, lemmatize, stemmer):\n",
        "    if remove_stopwords:\n",
        "        vectorizer = TfidfVectorizer(analyzer='word', input='content', stop_words=stopwords.words('english'))\n",
        "    else:\n",
        "        vectorizer =  TfidfVectorizer(analyzer='word', input='content')\n",
        "\n",
        "    clean_train = []\n",
        "    for paragraph in train:\n",
        "        paragraph_result = cleanText(paragraph, lemmatize, stemmer)\n",
        "        paragraph = \" \".join(str(x) for x in paragraph_result)\n",
        "        clean_train.append(paragraph)\n",
        "\n",
        "    clean_test = []\n",
        "    for paragraph in test:\n",
        "        paragraph_result = cleanText(paragraph, lemmatize, stemmer)\n",
        "        paragraph = \" \".join(str(x) for x in paragraph_result)\n",
        "        clean_test.append(paragraph)\n",
        "\n",
        "    tfidf_train = vectorizer.fit_transform(clean_train).toarray()\n",
        "    tfidf_test = vectorizer.transform(clean_test).toarray()\n",
        "    return tfidf_train, tfidf_test"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7g0jS45epcC5"
      },
      "source": [
        "# **Section 2: UNDERSTANDING THE DATA : A REVIEWS DATASET**\n",
        "\n",
        "Sentiment analysis is the interpretation and classification of emotions (such as positive, negative and neutral) within text data using text analysis techniques.  \n",
        "Given below is a dataset consisting of reviews along with sentiment class (positive or negative)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jU6875-NxrHw",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 73
        },
        "outputId": "4ade219b-fb5c-409a-e7c3-c2ec9d6d19a8"
      },
      "source": [
        "# Upload the Reviews CSV file that has been shared with you.\n",
        "# Run this cell, click on the 'Choose files' button and upload the file.\n",
        "from google.colab import files\n",
        "uploaded = files.upload()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "     <input type=\"file\" id=\"files-60d146f2-53ee-4323-842b-e140729af4af\" name=\"files[]\" multiple disabled\n",
              "        style=\"border:none\" />\n",
              "     <output id=\"result-60d146f2-53ee-4323-842b-e140729af4af\">\n",
              "      Upload widget is only available when the cell has been executed in the\n",
              "      current browser session. Please rerun this cell to enable.\n",
              "      </output>\n",
              "      <script>// Copyright 2017 Google LLC\n",
              "//\n",
              "// Licensed under the Apache License, Version 2.0 (the \"License\");\n",
              "// you may not use this file except in compliance with the License.\n",
              "// You may obtain a copy of the License at\n",
              "//\n",
              "//      http://www.apache.org/licenses/LICENSE-2.0\n",
              "//\n",
              "// Unless required by applicable law or agreed to in writing, software\n",
              "// distributed under the License is distributed on an \"AS IS\" BASIS,\n",
              "// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
              "// See the License for the specific language governing permissions and\n",
              "// limitations under the License.\n",
              "\n",
              "/**\n",
              " * @fileoverview Helpers for google.colab Python module.\n",
              " */\n",
              "(function(scope) {\n",
              "function span(text, styleAttributes = {}) {\n",
              "  const element = document.createElement('span');\n",
              "  element.textContent = text;\n",
              "  for (const key of Object.keys(styleAttributes)) {\n",
              "    element.style[key] = styleAttributes[key];\n",
              "  }\n",
              "  return element;\n",
              "}\n",
              "\n",
              "// Max number of bytes which will be uploaded at a time.\n",
              "const MAX_PAYLOAD_SIZE = 100 * 1024;\n",
              "\n",
              "function _uploadFiles(inputId, outputId) {\n",
              "  const steps = uploadFilesStep(inputId, outputId);\n",
              "  const outputElement = document.getElementById(outputId);\n",
              "  // Cache steps on the outputElement to make it available for the next call\n",
              "  // to uploadFilesContinue from Python.\n",
              "  outputElement.steps = steps;\n",
              "\n",
              "  return _uploadFilesContinue(outputId);\n",
              "}\n",
              "\n",
              "// This is roughly an async generator (not supported in the browser yet),\n",
              "// where there are multiple asynchronous steps and the Python side is going\n",
              "// to poll for completion of each step.\n",
              "// This uses a Promise to block the python side on completion of each step,\n",
              "// then passes the result of the previous step as the input to the next step.\n",
              "function _uploadFilesContinue(outputId) {\n",
              "  const outputElement = document.getElementById(outputId);\n",
              "  const steps = outputElement.steps;\n",
              "\n",
              "  const next = steps.next(outputElement.lastPromiseValue);\n",
              "  return Promise.resolve(next.value.promise).then((value) => {\n",
              "    // Cache the last promise value to make it available to the next\n",
              "    // step of the generator.\n",
              "    outputElement.lastPromiseValue = value;\n",
              "    return next.value.response;\n",
              "  });\n",
              "}\n",
              "\n",
              "/**\n",
              " * Generator function which is called between each async step of the upload\n",
              " * process.\n",
              " * @param {string} inputId Element ID of the input file picker element.\n",
              " * @param {string} outputId Element ID of the output display.\n",
              " * @return {!Iterable<!Object>} Iterable of next steps.\n",
              " */\n",
              "function* uploadFilesStep(inputId, outputId) {\n",
              "  const inputElement = document.getElementById(inputId);\n",
              "  inputElement.disabled = false;\n",
              "\n",
              "  const outputElement = document.getElementById(outputId);\n",
              "  outputElement.innerHTML = '';\n",
              "\n",
              "  const pickedPromise = new Promise((resolve) => {\n",
              "    inputElement.addEventListener('change', (e) => {\n",
              "      resolve(e.target.files);\n",
              "    });\n",
              "  });\n",
              "\n",
              "  const cancel = document.createElement('button');\n",
              "  inputElement.parentElement.appendChild(cancel);\n",
              "  cancel.textContent = 'Cancel upload';\n",
              "  const cancelPromise = new Promise((resolve) => {\n",
              "    cancel.onclick = () => {\n",
              "      resolve(null);\n",
              "    };\n",
              "  });\n",
              "\n",
              "  // Wait for the user to pick the files.\n",
              "  const files = yield {\n",
              "    promise: Promise.race([pickedPromise, cancelPromise]),\n",
              "    response: {\n",
              "      action: 'starting',\n",
              "    }\n",
              "  };\n",
              "\n",
              "  cancel.remove();\n",
              "\n",
              "  // Disable the input element since further picks are not allowed.\n",
              "  inputElement.disabled = true;\n",
              "\n",
              "  if (!files) {\n",
              "    return {\n",
              "      response: {\n",
              "        action: 'complete',\n",
              "      }\n",
              "    };\n",
              "  }\n",
              "\n",
              "  for (const file of files) {\n",
              "    const li = document.createElement('li');\n",
              "    li.append(span(file.name, {fontWeight: 'bold'}));\n",
              "    li.append(span(\n",
              "        `(${file.type || 'n/a'}) - ${file.size} bytes, ` +\n",
              "        `last modified: ${\n",
              "            file.lastModifiedDate ? file.lastModifiedDate.toLocaleDateString() :\n",
              "                                    'n/a'} - `));\n",
              "    const percent = span('0% done');\n",
              "    li.appendChild(percent);\n",
              "\n",
              "    outputElement.appendChild(li);\n",
              "\n",
              "    const fileDataPromise = new Promise((resolve) => {\n",
              "      const reader = new FileReader();\n",
              "      reader.onload = (e) => {\n",
              "        resolve(e.target.result);\n",
              "      };\n",
              "      reader.readAsArrayBuffer(file);\n",
              "    });\n",
              "    // Wait for the data to be ready.\n",
              "    let fileData = yield {\n",
              "      promise: fileDataPromise,\n",
              "      response: {\n",
              "        action: 'continue',\n",
              "      }\n",
              "    };\n",
              "\n",
              "    // Use a chunked sending to avoid message size limits. See b/62115660.\n",
              "    let position = 0;\n",
              "    do {\n",
              "      const length = Math.min(fileData.byteLength - position, MAX_PAYLOAD_SIZE);\n",
              "      const chunk = new Uint8Array(fileData, position, length);\n",
              "      position += length;\n",
              "\n",
              "      const base64 = btoa(String.fromCharCode.apply(null, chunk));\n",
              "      yield {\n",
              "        response: {\n",
              "          action: 'append',\n",
              "          file: file.name,\n",
              "          data: base64,\n",
              "        },\n",
              "      };\n",
              "\n",
              "      let percentDone = fileData.byteLength === 0 ?\n",
              "          100 :\n",
              "          Math.round((position / fileData.byteLength) * 100);\n",
              "      percent.textContent = `${percentDone}% done`;\n",
              "\n",
              "    } while (position < fileData.byteLength);\n",
              "  }\n",
              "\n",
              "  // All done.\n",
              "  yield {\n",
              "    response: {\n",
              "      action: 'complete',\n",
              "    }\n",
              "  };\n",
              "}\n",
              "\n",
              "scope.google = scope.google || {};\n",
              "scope.google.colab = scope.google.colab || {};\n",
              "scope.google.colab._files = {\n",
              "  _uploadFiles,\n",
              "  _uploadFilesContinue,\n",
              "};\n",
              "})(self);\n",
              "</script> "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving reviews.csv to reviews.csv\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "df = pd.read_csv('reviews.csv')"
      ],
      "metadata": {
        "id": "HILJMpa_y26e"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df = df.dropna()"
      ],
      "metadata": {
        "id": "WXD0iAR5k62v"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df.to_csv('reviews.csv', index=False)"
      ],
      "metadata": {
        "id": "7j9rNGkQpn7y"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-QXo11Bxvytu"
      },
      "source": [
        "# **Section 3: KNN MODEL**\n",
        "\n",
        "Given below are two KNN models; in the first case we are using Bag-of-Words and in the second case we are using TF-IDF.\n",
        "Note the different metrics and parameters used in each."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "R5yZXboZ92Z4"
      },
      "source": [
        "from sklearn import metrics, neighbors\n",
        "from sklearn.model_selection import train_test_split, cross_val_score, cross_val_predict\n",
        "\n",
        "## TASK - 1: Tweak the models below and see results with different parameters and distance metrics.\n",
        "\n",
        "def bow_knn():\n",
        "    \"\"\"Method for determining nearest neighbors using bag-of-words and K-Nearest Neighbor algorithm\"\"\"\n",
        "\n",
        "    training_data = pd.read_csv('reviews.csv')\n",
        "    X_train, X_test, y_train, y_test = train_test_split(training_data[\"sentence\"], training_data[\"sentiment\"], test_size=0.2, random_state=5)\n",
        "    X_train, X_test = createBagOfWords(X_train, X_test, remove_stopwords=True, lemmatize=True, stemmer=False)\n",
        "    # print(X_train)\n",
        "    knn = neighbors.KNeighborsClassifier(n_neighbors=5, weights='uniform', algorithm='auto', leaf_size=30, p=2, metric='manhattan', metric_params=None, n_jobs=1)\n",
        "\n",
        "    knn.fit(X_train, y_train)\n",
        "    predicted = knn.predict(X_test)\n",
        "    acc = metrics.accuracy_score(y_test, predicted)\n",
        "    print('KNN with BOW accuracy = ' + str(acc * 100) + '%')\n",
        "\n",
        "    scores = cross_val_score(knn, X_train, y_train, cv=3)\n",
        "    print(\"Cross Validation Accuracy: %0.2f\" % (scores.mean()))\n",
        "    print(scores)\n",
        "    print('\\n')\n",
        "    return predicted, y_test\n",
        "\n",
        "\n",
        "def tfidf_knn():\n",
        "    \"\"\"Method for determining nearest neighbors using tf-idf and K-Nearest Neighbor algorithm\"\"\"\n",
        "\n",
        "    training_data = pd.read_csv('reviews.csv')\n",
        "    X_train, X_test, y_train, y_test = train_test_split(training_data[\"sentence\"], training_data[\"sentiment\"],\n",
        "                                                        test_size=0.2, random_state=5)\n",
        "    X_train, X_test = createTFIDF(X_train, X_test, remove_stopwords=True, lemmatize=True, stemmer=False)\n",
        "    # print(X_train)\n",
        "    knn = neighbors.KNeighborsClassifier(n_neighbors=5, weights='distance', algorithm='brute', leaf_size=30, p=2,\n",
        "                                         metric='cosine', metric_params=None, n_jobs=1)\n",
        "\n",
        "    knn.fit(X_train, y_train)\n",
        "    predicted = knn.predict(X_test)\n",
        "    acc = metrics.accuracy_score(y_test, predicted)\n",
        "    print('KNN with TFIDF accuracy = ' + str(acc * 100) + '%')\n",
        "\n",
        "    scores = cross_val_score(knn, X_train, y_train, cv=3)\n",
        "    print(\"Cross Validation Accuracy: %0.2f\" % (scores.mean()))\n",
        "    print(scores)\n",
        "    return predicted, y_test"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Note: Cross-validation will be discussed in detail in the upcoming lab session."
      ],
      "metadata": {
        "id": "uPuI3wrL-8ZJ"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Xzno1rRNHoFT",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8a45e624-e25e-423e-bb61-0b3bb6f5a26c"
      },
      "source": [
        "## KNN accuracy after using BoW\n",
        "predicted, y_test = bow_knn()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "KNN with BOW accuracy = 62.30366492146597%\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.8/dist-packages/sklearn/model_selection/_split.py:676: UserWarning: The least populated class in y has only 1 members, which is less than n_splits=3.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cross Validation Accuracy: 0.62\n",
            "[0.63137255 0.59607843 0.61811024]\n",
            "\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FI5NMP8L-4eW",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7aa3c8ff-0c23-4b8d-bc14-4f6faac82ca8"
      },
      "source": [
        "## KNN accuracy after using TFIDF\n",
        "predicted, y_test = tfidf_knn()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "KNN with TFIDF accuracy = 70.15706806282722%\n",
            "Cross Validation Accuracy: 0.73\n",
            "[0.7254902  0.74117647 0.72834646]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.8/dist-packages/sklearn/model_selection/_split.py:676: UserWarning: The least populated class in y has only 1 members, which is less than n_splits=3.\n",
            "  warnings.warn(\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vm2vYDVfa5AP"
      },
      "source": [
        "# Section 4: SPAM TEXT DATASET\n",
        "Now let's use what we've learnt to classify texts as spam or not spam."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 73
        },
        "id": "c1GZgvXCsKX1",
        "outputId": "de82f8a4-0a1c-485f-c341-bfcc2d5e5236"
      },
      "source": [
        "# Upload the spam text data CSV file that has been shared with you. You can also download the file from https://www.kaggle.com/datasets/uciml/sms-spam-collection-dataset\n",
        "# Run this cell, click on the 'Choose files' button and upload the file.\n",
        "from google.colab import files\n",
        "uploaded = files.upload()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "     <input type=\"file\" id=\"files-be30aeca-e603-49d4-8fd6-042f12753f6b\" name=\"files[]\" multiple disabled\n",
              "        style=\"border:none\" />\n",
              "     <output id=\"result-be30aeca-e603-49d4-8fd6-042f12753f6b\">\n",
              "      Upload widget is only available when the cell has been executed in the\n",
              "      current browser session. Please rerun this cell to enable.\n",
              "      </output>\n",
              "      <script>// Copyright 2017 Google LLC\n",
              "//\n",
              "// Licensed under the Apache License, Version 2.0 (the \"License\");\n",
              "// you may not use this file except in compliance with the License.\n",
              "// You may obtain a copy of the License at\n",
              "//\n",
              "//      http://www.apache.org/licenses/LICENSE-2.0\n",
              "//\n",
              "// Unless required by applicable law or agreed to in writing, software\n",
              "// distributed under the License is distributed on an \"AS IS\" BASIS,\n",
              "// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
              "// See the License for the specific language governing permissions and\n",
              "// limitations under the License.\n",
              "\n",
              "/**\n",
              " * @fileoverview Helpers for google.colab Python module.\n",
              " */\n",
              "(function(scope) {\n",
              "function span(text, styleAttributes = {}) {\n",
              "  const element = document.createElement('span');\n",
              "  element.textContent = text;\n",
              "  for (const key of Object.keys(styleAttributes)) {\n",
              "    element.style[key] = styleAttributes[key];\n",
              "  }\n",
              "  return element;\n",
              "}\n",
              "\n",
              "// Max number of bytes which will be uploaded at a time.\n",
              "const MAX_PAYLOAD_SIZE = 100 * 1024;\n",
              "\n",
              "function _uploadFiles(inputId, outputId) {\n",
              "  const steps = uploadFilesStep(inputId, outputId);\n",
              "  const outputElement = document.getElementById(outputId);\n",
              "  // Cache steps on the outputElement to make it available for the next call\n",
              "  // to uploadFilesContinue from Python.\n",
              "  outputElement.steps = steps;\n",
              "\n",
              "  return _uploadFilesContinue(outputId);\n",
              "}\n",
              "\n",
              "// This is roughly an async generator (not supported in the browser yet),\n",
              "// where there are multiple asynchronous steps and the Python side is going\n",
              "// to poll for completion of each step.\n",
              "// This uses a Promise to block the python side on completion of each step,\n",
              "// then passes the result of the previous step as the input to the next step.\n",
              "function _uploadFilesContinue(outputId) {\n",
              "  const outputElement = document.getElementById(outputId);\n",
              "  const steps = outputElement.steps;\n",
              "\n",
              "  const next = steps.next(outputElement.lastPromiseValue);\n",
              "  return Promise.resolve(next.value.promise).then((value) => {\n",
              "    // Cache the last promise value to make it available to the next\n",
              "    // step of the generator.\n",
              "    outputElement.lastPromiseValue = value;\n",
              "    return next.value.response;\n",
              "  });\n",
              "}\n",
              "\n",
              "/**\n",
              " * Generator function which is called between each async step of the upload\n",
              " * process.\n",
              " * @param {string} inputId Element ID of the input file picker element.\n",
              " * @param {string} outputId Element ID of the output display.\n",
              " * @return {!Iterable<!Object>} Iterable of next steps.\n",
              " */\n",
              "function* uploadFilesStep(inputId, outputId) {\n",
              "  const inputElement = document.getElementById(inputId);\n",
              "  inputElement.disabled = false;\n",
              "\n",
              "  const outputElement = document.getElementById(outputId);\n",
              "  outputElement.innerHTML = '';\n",
              "\n",
              "  const pickedPromise = new Promise((resolve) => {\n",
              "    inputElement.addEventListener('change', (e) => {\n",
              "      resolve(e.target.files);\n",
              "    });\n",
              "  });\n",
              "\n",
              "  const cancel = document.createElement('button');\n",
              "  inputElement.parentElement.appendChild(cancel);\n",
              "  cancel.textContent = 'Cancel upload';\n",
              "  const cancelPromise = new Promise((resolve) => {\n",
              "    cancel.onclick = () => {\n",
              "      resolve(null);\n",
              "    };\n",
              "  });\n",
              "\n",
              "  // Wait for the user to pick the files.\n",
              "  const files = yield {\n",
              "    promise: Promise.race([pickedPromise, cancelPromise]),\n",
              "    response: {\n",
              "      action: 'starting',\n",
              "    }\n",
              "  };\n",
              "\n",
              "  cancel.remove();\n",
              "\n",
              "  // Disable the input element since further picks are not allowed.\n",
              "  inputElement.disabled = true;\n",
              "\n",
              "  if (!files) {\n",
              "    return {\n",
              "      response: {\n",
              "        action: 'complete',\n",
              "      }\n",
              "    };\n",
              "  }\n",
              "\n",
              "  for (const file of files) {\n",
              "    const li = document.createElement('li');\n",
              "    li.append(span(file.name, {fontWeight: 'bold'}));\n",
              "    li.append(span(\n",
              "        `(${file.type || 'n/a'}) - ${file.size} bytes, ` +\n",
              "        `last modified: ${\n",
              "            file.lastModifiedDate ? file.lastModifiedDate.toLocaleDateString() :\n",
              "                                    'n/a'} - `));\n",
              "    const percent = span('0% done');\n",
              "    li.appendChild(percent);\n",
              "\n",
              "    outputElement.appendChild(li);\n",
              "\n",
              "    const fileDataPromise = new Promise((resolve) => {\n",
              "      const reader = new FileReader();\n",
              "      reader.onload = (e) => {\n",
              "        resolve(e.target.result);\n",
              "      };\n",
              "      reader.readAsArrayBuffer(file);\n",
              "    });\n",
              "    // Wait for the data to be ready.\n",
              "    let fileData = yield {\n",
              "      promise: fileDataPromise,\n",
              "      response: {\n",
              "        action: 'continue',\n",
              "      }\n",
              "    };\n",
              "\n",
              "    // Use a chunked sending to avoid message size limits. See b/62115660.\n",
              "    let position = 0;\n",
              "    do {\n",
              "      const length = Math.min(fileData.byteLength - position, MAX_PAYLOAD_SIZE);\n",
              "      const chunk = new Uint8Array(fileData, position, length);\n",
              "      position += length;\n",
              "\n",
              "      const base64 = btoa(String.fromCharCode.apply(null, chunk));\n",
              "      yield {\n",
              "        response: {\n",
              "          action: 'append',\n",
              "          file: file.name,\n",
              "          data: base64,\n",
              "        },\n",
              "      };\n",
              "\n",
              "      let percentDone = fileData.byteLength === 0 ?\n",
              "          100 :\n",
              "          Math.round((position / fileData.byteLength) * 100);\n",
              "      percent.textContent = `${percentDone}% done`;\n",
              "\n",
              "    } while (position < fileData.byteLength);\n",
              "  }\n",
              "\n",
              "  // All done.\n",
              "  yield {\n",
              "    response: {\n",
              "      action: 'complete',\n",
              "    }\n",
              "  };\n",
              "}\n",
              "\n",
              "scope.google = scope.google || {};\n",
              "scope.google.colab = scope.google.colab || {};\n",
              "scope.google.colab._files = {\n",
              "  _uploadFiles,\n",
              "  _uploadFilesContinue,\n",
              "};\n",
              "})(self);\n",
              "</script> "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving spam.csv to spam.csv\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 513
        },
        "id": "qRiS7dT7piTE",
        "outputId": "0017a750-48f4-43c5-b22b-cb9c31db6614"
      },
      "source": [
        "import pandas as pd\n",
        "df = pd.read_csv('spam.csv', error_bad_lines=False)\n",
        "df"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.8/dist-packages/IPython/core/interactiveshell.py:3326: FutureWarning: The error_bad_lines argument has been deprecated and will be removed in a future version.\n",
            "\n",
            "\n",
            "  exec(code_obj, self.user_global_ns, self.user_ns)\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     Category                                            Message\n",
              "0         ham  Go until jurong point, crazy.. Available only ...\n",
              "1         ham                      Ok lar... Joking wif u oni...\n",
              "2        spam  Free entry in 2 a wkly comp to win FA Cup fina...\n",
              "3         ham  U dun say so early hor... U c already then say...\n",
              "4         ham  Nah I don't think he goes to usf, he lives aro...\n",
              "...       ...                                                ...\n",
              "5567     spam  This is the 2nd time we have tried 2 contact u...\n",
              "5568      ham              Will Ã¼ b going to esplanade fr home?\n",
              "5569      ham  Pity, * was in mood for that. So...any other s...\n",
              "5570      ham  The guy did some bitching but I acted like i'd...\n",
              "5571      ham                         Rofl. Its true to its name\n",
              "\n",
              "[5572 rows x 2 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-f0118d50-26fe-4af9-b6f1-de6a56b37134\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Category</th>\n",
              "      <th>Message</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ham</td>\n",
              "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ham</td>\n",
              "      <td>Ok lar... Joking wif u oni...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>spam</td>\n",
              "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>ham</td>\n",
              "      <td>U dun say so early hor... U c already then say...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>ham</td>\n",
              "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5567</th>\n",
              "      <td>spam</td>\n",
              "      <td>This is the 2nd time we have tried 2 contact u...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5568</th>\n",
              "      <td>ham</td>\n",
              "      <td>Will Ã¼ b going to esplanade fr home?</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5569</th>\n",
              "      <td>ham</td>\n",
              "      <td>Pity, * was in mood for that. So...any other s...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5570</th>\n",
              "      <td>ham</td>\n",
              "      <td>The guy did some bitching but I acted like i'd...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5571</th>\n",
              "      <td>ham</td>\n",
              "      <td>Rofl. Its true to its name</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5572 rows × 2 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-f0118d50-26fe-4af9-b6f1-de6a56b37134')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-f0118d50-26fe-4af9-b6f1-de6a56b37134 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-f0118d50-26fe-4af9-b6f1-de6a56b37134');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OsvBm4luNCME"
      },
      "source": [
        "df['Category'] = df['Category'].map({'ham': 0, 'spam': 1})"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "KiEHuMypWyb6",
        "outputId": "cc69f16c-fc89-4dbc-e0fe-7c602401cf04"
      },
      "source": [
        "df.head(5)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   Category                                            Message\n",
              "0         0  Go until jurong point, crazy.. Available only ...\n",
              "1         0                      Ok lar... Joking wif u oni...\n",
              "2         1  Free entry in 2 a wkly comp to win FA Cup fina...\n",
              "3         0  U dun say so early hor... U c already then say...\n",
              "4         0  Nah I don't think he goes to usf, he lives aro..."
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-fbfc5e8c-0911-427a-b858-65e4e0d9fe2e\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Category</th>\n",
              "      <th>Message</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0</td>\n",
              "      <td>Ok lar... Joking wif u oni...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1</td>\n",
              "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0</td>\n",
              "      <td>U dun say so early hor... U c already then say...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0</td>\n",
              "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-fbfc5e8c-0911-427a-b858-65e4e0d9fe2e')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-fbfc5e8c-0911-427a-b858-65e4e0d9fe2e button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-fbfc5e8c-0911-427a-b858-65e4e0d9fe2e');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "len(df)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WRJU9rFy1XQR",
        "outputId": "a178e29b-fd49-40c9-fd70-aab1d5f4bb4f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "5572"
            ]
          },
          "metadata": {},
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wnv0v4T6sqJQ"
      },
      "source": [
        "from sklearn import metrics, neighbors\n",
        "from sklearn.model_selection import train_test_split, cross_val_score, cross_val_predict\n",
        "\n",
        "## TASK - 2: Tweak the models below and see results with different parameters and distance metrics.\n",
        "\n",
        "def bow_knn():\n",
        "    \"\"\"Method for determining nearest neighbors using bag-of-words and K-Nearest Neighbor algorithm\"\"\"\n",
        "\n",
        "    training_data = pd.read_csv('spam.csv')\n",
        "    training_data['Category'] = training_data['Category'].map({'ham': 0, 'spam': 1})\n",
        "    X_train, X_test, y_train, y_test = train_test_split(training_data[\"Message\"], training_data[\"Category\"], test_size=0.2, random_state=5)\n",
        "    X_train, X_test = createBagOfWords(X_train, X_test, remove_stopwords=True, lemmatize=True, stemmer=False)\n",
        "    knn = neighbors.KNeighborsClassifier(n_neighbors=5, weights='uniform', algorithm='auto', leaf_size=30, p=2, metric='cosine', metric_params=None, n_jobs=1)\n",
        "\n",
        "    knn.fit(X_train, y_train)\n",
        "    predicted = knn.predict(X_test)\n",
        "    acc = metrics.accuracy_score(y_test, predicted)\n",
        "    print('KNN with BOW accuracy = ' + str(acc * 100) + '%')\n",
        "\n",
        "    scores = cross_val_score(knn, X_train, y_train, cv=3)\n",
        "    print(\"Cross Validation Accuracy: %0.2f\" % (scores.mean()))\n",
        "    print(scores)\n",
        "    print('\\n')\n",
        "    return predicted, y_test\n",
        "\n",
        "\n",
        "def tfidf_knn():\n",
        "    \"\"\"Method for determining nearest neighbors using tf-idf and K-Nearest Neighbor algorithm\"\"\"\n",
        "\n",
        "    training_data = pd.read_csv('spam.csv')\n",
        "    training_data['Category'] = training_data['Category'].map({'ham': 0, 'spam': 1})\n",
        "    X_train, X_test, y_train, y_test = train_test_split(training_data[\"Message\"], training_data[\"Category\"], test_size=0.2, random_state=5)\n",
        "    X_train, X_test = createTFIDF(X_train, X_test, remove_stopwords=True, lemmatize=True, stemmer=False)\n",
        "    knn = neighbors.KNeighborsClassifier(n_neighbors=5, weights='distance', algorithm='brute', leaf_size=30, p=2, metric='euclidean', metric_params=None, n_jobs=1)\n",
        "\n",
        "    knn.fit(X_train, y_train)\n",
        "    predicted = knn.predict(X_test)\n",
        "    acc = metrics.accuracy_score(y_test, predicted)\n",
        "    print('KNN with TFIDF accuracy = ' + str(acc * 100) + '%')\n",
        "\n",
        "    scores = cross_val_score(knn, X_train, y_train, cv=3)\n",
        "    print(\"Cross Validation Accuracy: %0.2f\" % (scores.mean()))\n",
        "    print(scores)\n",
        "    return predicted, y_test"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "W8PwydHYs1h_",
        "outputId": "6a64b5c8-ef86-4b4e-d0a0-344bc8ba8e8c"
      },
      "source": [
        "# This cell may take some time to run\n",
        "predicted, y_test = bow_knn()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "KNN with BOW accuracy = 97.9372197309417%\n",
            "Cross Validation Accuracy: 0.96\n",
            "[0.95827725 0.95962315 0.96498316]\n",
            "\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# This cell may take some time to run\n",
        "predicted, y_test = tfidf_knn()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zf8i2P1nxpl8",
        "outputId": "096f465f-3256-426c-ecf1-971ae0a7a10e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "KNN with TFIDF accuracy = 93.99103139013452%\n",
            "Cross Validation Accuracy: 0.92\n",
            "[0.92664872 0.91857335 0.92525253]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Because of Different distance is used. Here, BoW have higher accuracy than TFIDF. \n",
        "# But out statement was, TFIDF always do better than BoW if all other perameters are common. So it holds good as we can see below."
      ],
      "metadata": {
        "id": "bq0O7w64ufqb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def bow_knn():\n",
        "    \"\"\"Method for determining nearest neighbors using bag-of-words and K-Nearest Neighbor algorithm\"\"\"\n",
        "\n",
        "    training_data = pd.read_csv('spam.csv')\n",
        "    training_data['Category'] = training_data['Category'].map({'ham': 0, 'spam': 1})\n",
        "    X_train, X_test, y_train, y_test = train_test_split(training_data[\"Message\"], training_data[\"Category\"], test_size=0.2, random_state=5)\n",
        "    X_train, X_test = createBagOfWords(X_train, X_test, remove_stopwords=True, lemmatize=True, stemmer=False)\n",
        "    knn = neighbors.KNeighborsClassifier(n_neighbors=5, weights='uniform', algorithm='auto', leaf_size=30, p=2, metric='euclidean', metric_params=None, n_jobs=1)\n",
        "\n",
        "    knn.fit(X_train, y_train)\n",
        "    predicted = knn.predict(X_test)\n",
        "    acc = metrics.accuracy_score(y_test, predicted)\n",
        "    print('KNN with BOW accuracy = ' + str(acc * 100) + '%')\n",
        "\n",
        "    scores = cross_val_score(knn, X_train, y_train, cv=3)\n",
        "    print(\"Cross Validation Accuracy: %0.2f\" % (scores.mean()))\n",
        "    print(scores)\n",
        "    print('\\n')\n",
        "    return predicted, y_test\n",
        "\n",
        "\n",
        "def tfidf_knn():\n",
        "    \"\"\"Method for determining nearest neighbors using tf-idf and K-Nearest Neighbor algorithm\"\"\"\n",
        "\n",
        "    training_data = pd.read_csv('spam.csv')\n",
        "    training_data['Category'] = training_data['Category'].map({'ham': 0, 'spam': 1})\n",
        "    X_train, X_test, y_train, y_test = train_test_split(training_data[\"Message\"], training_data[\"Category\"], test_size=0.2, random_state=5)\n",
        "    X_train, X_test = createTFIDF(X_train, X_test, remove_stopwords=True, lemmatize=True, stemmer=False)\n",
        "    knn = neighbors.KNeighborsClassifier(n_neighbors=5, weights='distance', algorithm='brute', leaf_size=30, p=2, metric='euclidean', metric_params=None, n_jobs=1)\n",
        "\n",
        "    knn.fit(X_train, y_train)\n",
        "    predicted = knn.predict(X_test)\n",
        "    acc = metrics.accuracy_score(y_test, predicted)\n",
        "    print('KNN with TFIDF accuracy = ' + str(acc * 100) + '%')\n",
        "\n",
        "    scores = cross_val_score(knn, X_train, y_train, cv=3)\n",
        "    print(\"Cross Validation Accuracy: %0.2f\" % (scores.mean()))\n",
        "    print(scores)\n",
        "    return predicted, y_test"
      ],
      "metadata": {
        "id": "DTvjvblvuhKO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "predicted, y_test = bow_knn()"
      ],
      "metadata": {
        "id": "qM53V8lkuuJC",
        "outputId": "d5b2d312-0475-4acb-a766-16912311e390",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "KNN with BOW accuracy = 92.19730941704036%\n",
            "Cross Validation Accuracy: 0.91\n",
            "[0.90713324 0.90040377 0.91245791]\n",
            "\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "predicted, y_test = tfidf_knn()"
      ],
      "metadata": {
        "id": "kySiepaxu-ef",
        "outputId": "c69c53db-0b83-4fd5-d679-5dadb66eff1f",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "KNN with TFIDF accuracy = 93.99103139013452%\n",
            "Cross Validation Accuracy: 0.92\n",
            "[0.92664872 0.91857335 0.92525253]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Questions to Think About and Answer\n",
        "1. Why does the TF-IDF approach generally result in a better accuracy than Bag-of-Words ?\n",
        "\n",
        "\n",
        "**ANSWER** :- Bag-of-Words representation only gives frequency of a particular word in document.TF-IDF counting is normalized, So large and small document have same representation. It gives importance to unique words. Which tells which are important words and which are less important word, that will be useful in classification.\n",
        "\n",
        "\n",
        "\n",
        "2. Can you think of techniques that are better than both BoW and TF-IDF ?\n",
        "\n",
        "**ANSWER**:-  BoW and TF-IDF doesn't capture meaning of the words. It cannot detect similartiy between two words. To do that we can use Word2Vec represetation Which can cluster similar word to one group. and also we can use * text processing steps and *Convolution neural network.\n",
        "\n",
        "\n",
        "\n",
        "3. Read about Stemming and Lemmatization from the resources given below. Think about the pros/cons of each.\n",
        "\n",
        "**ANSWER**:- \n",
        "\n",
        " **Stemming:**\n",
        "\n",
        "  It is straightforward to implement and runs fast. But it not give output in valid english form. It only care about similar words should be part of same stem.\n",
        "\n",
        "  **Lemmatization:**\n",
        "\n",
        "  It gives meanining full, valid English word as output.\n",
        "  But it is slower than stem as it need to check with real-world dictionary. "
      ],
      "metadata": {
        "id": "f9xvbzS4yLTr"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Useful Resources for further reading\n",
        "1. Stemming and Lemmatization: https://nlp.stanford.edu/IR-book/html/htmledition/stemming-and-lemmatization-1.html\n",
        "2. TF-IDF and BoW : https://www.analyticsvidhya.com/blog/2020/02/quick-introduction-bag-of-words-bow-tf-idf/\n",
        "3. TF-IDF: https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfTransformer.html\n"
      ],
      "metadata": {
        "id": "T6xPL6smyWG-"
      }
    }
  ]
}