PySpark__Mapping_x_FlatMapping_Exercise_1.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/saurater/ciencia_de_dados_pyspark/blob/main/PySpark__Mapping_x_FlatMapping_Exercise_1.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# PySpark - Mapping x FlatMapping de Arquivo Texto - Parte I\n",
        "\n",
        "## Adaptação do IGTI por Sam Faraday\n",
        "\n",
        "Teste Inicial no Databricks\n",
        "June 2022"
      ],
      "metadata": {
        "application/vnd.databricks.v1+cell": {
          "title": "",
          "showTitle": false,
          "inputWidgets": {},
          "nuid": "c233a29b-65a1-4708-b383-3e6398c2688d"
        },
        "id": "V7rpmWN2CMje"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pip install pyspark"
      ],
      "metadata": {
        "id": "p-Lyy7XEC7VF",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a445c537-15f2-440f-8b6c-d9ad5219b265"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting pyspark\n",
            "  Downloading pyspark-3.2.1.tar.gz (281.4 MB)\n",
            "\u001b[K     |████████████████████████████████| 281.4 MB 34 kB/s \n",
            "\u001b[?25hCollecting py4j==0.10.9.3\n",
            "  Downloading py4j-0.10.9.3-py2.py3-none-any.whl (198 kB)\n",
            "\u001b[K     |████████████████████████████████| 198 kB 48.9 MB/s \n",
            "\u001b[?25hBuilding wheels for collected packages: pyspark\n",
            "  Building wheel for pyspark (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pyspark: filename=pyspark-3.2.1-py2.py3-none-any.whl size=281853642 sha256=7a0fa8423e98862e0f8ede6624fd46a11f7f47e44da8b8adfce8897b1daa1c5a\n",
            "  Stored in directory: /root/.cache/pip/wheels/9f/f5/07/7cd8017084dce4e93e84e92efd1e1d5334db05f2e83bcef74f\n",
            "Successfully built pyspark\n",
            "Installing collected packages: py4j, pyspark\n",
            "Successfully installed py4j-0.10.9.3 pyspark-3.2.1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from pyspark.sql import SparkSession\n",
        "spark  =  SparkSession.builder.appName(\"Mapeando_um_Arquivo\").getOrCreate()"
      ],
      "metadata": {
        "application/vnd.databricks.v1+cell": {
          "title": "Importa e Inicia uma Sessão",
          "showTitle": true,
          "inputWidgets": {},
          "nuid": "e3cbe37e-eb71-4154-9e36-b0da7be52ed5"
        },
        "id": "emDOhokjCMji"
      },
      "outputs": [],
      "execution_count": 2
    },
    {
      "cell_type": "code",
      "source": [
        "meu_arquivo_read_me = spark.sparkContext.textFile(\"SPARK_README.md\")"
      ],
      "metadata": {
        "application/vnd.databricks.v1+cell": {
          "title": "Carrega o arquivo SPARK_READ_.MD - vc precisa fazer o upload antes no Databricks",
          "showTitle": true,
          "inputWidgets": {},
          "nuid": "003f2dd0-2c7c-46a2-94bf-a70618046cfe"
        },
        "id": "z9B_k-JUCMjk"
      },
      "outputs": [],
      "execution_count": 3
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Mapping ou FlatMapping \n"
      ],
      "metadata": {
        "id": "onD3RSbIODES"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Para ver a diferença entre map e flatMap, comente essa linha , executando a próxima e vá até a impressão"
      ],
      "metadata": {
        "id": "5TDME8HzOiDi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Usanda uma função lambda, para cada linha ele quebra (split) por palavra -Atenção aos comentários abaixo\n",
        "#mapa_do_meu_arquivo_read_me = meu_arquivo_read_me.map(lambda linha_do_texto : linha_do_texto.split()) \n"
      ],
      "metadata": {
        "application/vnd.databricks.v1+cell": {
          "title": "Usanda uma função lambda, para cada linha ele adiciona o len (cumprimento) da linha",
          "showTitle": true,
          "inputWidgets": {},
          "nuid": "ede42bbb-e102-4c1a-8f96-d52633245cf1"
        },
        "id": "U5eKJt3gCMjk"
      },
      "outputs": [],
      "execution_count": 4
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Para ver a diferença entre map e flatMap, comente essa linha , executando a anterior e vá até a impressão"
      ],
      "metadata": {
        "id": "iiTFWSNpOqcR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "mapa_do_meu_arquivo_read_me = meu_arquivo_read_me.flatMap(lambda linha_do_texto : linha_do_texto.split())"
      ],
      "metadata": {
        "id": "wZqPN38WOnup"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Imprimindo no Workers\n",
        "## Nada será immpresso na tela aqui"
      ],
      "metadata": {
        "id": "lVqmI6bbO2Lp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Nada será impresso em tela. Apenas nos workers, por isso as duas linhas alternativas seguintes\n",
        "mapa_do_meu_arquivo_read_me.foreach(print)"
      ],
      "metadata": {
        "application/vnd.databricks.v1+cell": {
          "title": "Atenção: Não será impresso nenhum resultado em tela, apenas nos workers.  As 2 linhas seguintes são uma alternativa",
          "showTitle": true,
          "inputWidgets": {},
          "nuid": "756a1617-be8b-453d-84f2-6a8728b99ecf"
        },
        "id": "DiAQTtd7CMjk"
      },
      "outputs": [],
      "execution_count": 6
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Alternativa de impressão em tela"
      ],
      "metadata": {
        "id": "H7DQswy_PDMR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Pega-se os dados dos Workers e tra-se para o Master com a função collect\n",
        "saida_mapeada = mapa_do_meu_arquivo_read_me.collect()"
      ],
      "metadata": {
        "application/vnd.databricks.v1+cell": {
          "title": "Trago os dados dos Workders para o Master, colocando na variável saida_mapeada",
          "showTitle": true,
          "inputWidgets": {},
          "nuid": "085cd7a7-384f-44a7-8a03-92532fdeef8b"
        },
        "id": "jcqp2ym4CMjl"
      },
      "outputs": [],
      "execution_count": 7
    },
    {
      "cell_type": "code",
      "source": [
        "#imprime linha a linha, mostrando o tamanho de cada uma\n",
        "for cada_linha  in saida_mapeada:\n",
        "        print(cada_linha)\n",
        "       "
      ],
      "metadata": {
        "application/vnd.databricks.v1+cell": {
          "title": "Imprimo linha a linha da variável saida_mapeada",
          "showTitle": true,
          "inputWidgets": {},
          "nuid": "789af03b-42bd-4d18-9089-f73d51f2e613"
        },
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5I2PTKt-CMjl",
        "outputId": "9f0eb7fb-fedc-481e-e8b3-55b357e37e33"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Apache\n",
            "Spark\n",
            "Spark\n",
            "is\n",
            "a\n",
            "unified\n",
            "analytics\n",
            "engine\n",
            "for\n",
            "large-scale\n",
            "data\n",
            "processing.\n",
            "It\n",
            "provides\n",
            "high-level\n",
            "APIs\n",
            "in\n",
            "Scala,\n",
            "Java,\n",
            "Python,\n",
            "and\n",
            "R,\n",
            "and\n",
            "an\n",
            "optimized\n",
            "engine\n",
            "that\n",
            "supports\n",
            "general\n",
            "computation\n",
            "graphs\n",
            "for\n",
            "data\n",
            "analysis.\n",
            "It\n",
            "also\n",
            "supports\n",
            "a\n",
            "rich\n",
            "set\n",
            "of\n",
            "higher-level\n",
            "tools\n",
            "including\n",
            "Spark\n",
            "SQL\n",
            "for\n",
            "SQL\n",
            "and\n",
            "DataFrames,\n",
            "pandas\n",
            "API\n",
            "on\n",
            "Spark\n",
            "for\n",
            "pandas\n",
            "workloads,\n",
            "MLlib\n",
            "for\n",
            "machine\n",
            "learning,\n",
            "GraphX\n",
            "for\n",
            "graph\n",
            "processing,\n",
            "and\n",
            "Structured\n",
            "Streaming\n",
            "for\n",
            "stream\n",
            "processing.\n",
            "Online\n",
            "Documentation\n",
            "You\n",
            "can\n",
            "find\n",
            "the\n",
            "latest\n",
            "Spark\n",
            "documentation,\n",
            "including\n",
            "a\n",
            "programming\n",
            "guide,\n",
            "on\n",
            "the\n",
            "project\n",
            "web\n",
            "page.\n",
            "This\n",
            "README\n",
            "file\n",
            "only\n",
            "contains\n",
            "basic\n",
            "setup\n",
            "instructions.\n"
          ]
        }
      ],
      "execution_count": 8
    }
  ],
  "metadata": {
    "application/vnd.databricks.v1+notebook": {
      "notebookName": "Mapping_Exercise_1",
      "dashboards": [],
      "notebookMetadata": {
        "pythonIndentUnit": 4
      },
      "language": "python",
      "widgets": {},
      "notebookOrigID": 3279161451786732
    },
    "colab": {
      "name": "PySpark _Mapping_x_FlatMapping_Exercise_1.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "language_info": {
      "name": "python"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}