zarr_cloud_native_geospatial_2022_blank.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a2416980-93eb-4c03-ad75-f75be23e596d",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Zarr Tutorial\n",
    "\n",
    "👋 Welcome to the Zarr tutorial! This tutorial was created for the 2022 [Cloud-Native Geo Event](https://schedule.cloudnativegeo.org/). It borrows heavily from the [tutorial section of the Zarr docs](https://zarr.readthedocs.io/en/stable/tutorial.html).\n",
    "\n",
    "_Level: intermediate python programs. Ideally you are already a bit familiar with Numpy and Xarray._\n",
    "\n",
    "This will be a live-coding tutorial. You will get the most out of the tutorial if you follow along in a blank notebook and type the code in yourself. \n",
    "\n",
    "**Learning Goals**\n",
    "\n",
    "By the end of this tutorial, you should be able to \n",
    "\n",
    "- Identify the fundamental data structures in Zarr (Groups and Arrays) and the key properties of Arrays (shape, dtype, chunks, attributes)\n",
    "- Create Arrays and Groups in local files or in S3\n",
    "- Create and edit attributes (metadata)\n",
    "- Read and write data into Arrays\n",
    "- Evaluate the tradeoffs of different Array chunking strageies\n",
    "- Read and write NetCDF-style data to Zarr using Xarray\n",
    "- Do parallel processing on Xarray / Zarr data using Dask"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4e8912c-6cdb-4e9c-af22-fc2071047887",
   "metadata": {},
   "source": [
    "## Review: NumPy\n",
    "\n",
    "Zarr's python API borrows many concepts from NumPy. So first we review the basics of NumPy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d89cb0b-f3cd-4ff2-8c97-5e4db5e60165",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "a2cee034-30de-474c-b53c-da580a30b3db",
   "metadata": {},
   "source": [
    "Creating an array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d530a6f-7ea8-461d-ae67-496f45c8cb98",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "ecb9ff0d-30f0-47a6-b5c9-6b4488f4e7aa",
   "metadata": {},
   "source": [
    "This array lives in memory.\n",
    "\n",
    "How much memory does the array use?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e28c892d-12c2-4f01-9a15-16addd62a5da",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "e1dfb3cb-c67d-43bf-bddd-23b15d939148",
   "metadata": {},
   "source": [
    "Getting a piece of data with slicing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b8c2e92-f76d-4c99-9f95-11a44b146da6",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "1be8ddc6-c6a8-4ecf-905b-acb11ad36427",
   "metadata": {},
   "source": [
    "Create a new array and assign to it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c831a4a-871e-4584-bd28-09a835607445",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f60f7c8-a4fb-4fce-aa6c-1de03b4c2a7d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "46a9aa21-acf2-4327-8deb-04d4b3ea05ae",
   "metadata": {},
   "source": [
    "## Zarr Fundamentals\n",
    "\n",
    "A Zarr array has three important properties: \n",
    "- Shape\n",
    "- Dtype\n",
    "- Attributes\n",
    "\n",
    "An additional property of arrays we will not delve into here is _filters / compressors_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a050e15-ae6e-470d-b13c-f63e96e95623",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c52a2c9d-6281-4a0a-9e34-1cd90c96a8f7",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6c4f256-8cdd-477c-b1cf-81ffefd8a972",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "df77a44f-2aae-40ba-a26a-cfc5ab82ee34",
   "metadata": {},
   "source": [
    "No data has been written to the array yet. If we try to access data, we will just get the fill value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4dd6aeb5-5c5e-4e02-b99d-ee07d1ee6992",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "684627cc-6eea-4e86-bd68-83b68f2df8f7",
   "metadata": {},
   "source": [
    "This is how we assign data to the array. When we do this it gets written immediately."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f238a89-cc5e-460c-82bb-4a67442dbcd5",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37d2f94a-e9c5-4627-a11f-60741374abc2",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "313fe85b-125e-4b30-a284-912b03ad05e4",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Attributes\n",
    "\n",
    "We can attach arbitrary metadata to our Array via attributes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "963fd2eb-efe8-49a9-be31-be26730a5c46",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c0fed1c-9608-4260-9beb-0f837de31826",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "8eed49a7-5c6a-4dc1-aee7-4563daa23791",
   "metadata": {},
   "source": [
    "### Under the hood\n",
    "\n",
    "Where / how is our data actually stored?\n",
    "Let's look under the hood. _The ability to look inside a Zarr store and understand what is there is a deliberate design decision._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e140512-20d3-4d03-9acf-422423d8e053",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cd8c7b5c-ac7f-4553-8769-c97760dd433f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a0816b9d-4ec8-4e02-81b5-29d3635bb1cc",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "485e2490-c282-4e78-a644-741c7f589603",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "17fb5bc8-9a69-4daa-aa75-99928ed6b104",
   "metadata": {},
   "source": [
    "### Choosing Chunks\n",
    "\n",
    "The main parameter we control when creating Zarr Arrays is the chunk shape.\n",
    "When selecting chunks, we need to keep in mind two constraints:\n",
    "- Writes can be concurrent (come from different processes simultaneously) if they do not touch the same chunks. _This enables massively parallel writing in the cloud._\n",
    "- When reading data, if any piece of the chunk is needed, the entire chunk has to be loaded. (This will be relaxed in Zarr V3.)\n",
    "\n",
    "Here we will compare two different chunking stragegies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f1b04df-8be4-4203-95ff-a8fdf7d23bac",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "930dd0ff-3909-4ead-9490-cae8c37b9636",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4144c410-6133-4eea-8915-07fe3876360d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea2d0324-aded-4925-89ae-175fc3cdb72b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "26142bb9-23bf-45b8-974d-57ddc24ef1f0",
   "metadata": {},
   "source": [
    "There is no universally perfect chunk size / shape.\n",
    "Need to consider:\n",
    "- Access patterns for data\n",
    "- Latency & throughput of storage device\n",
    "- Constraints on number of files / objects (don't want a billion files)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78503d31-9bfa-4873-9da7-d1ce8a65abee",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Resizing Arrays\n",
    "\n",
    "It is trivial to change the shape of Zarr arrays.\n",
    "If you make your array smaller, you will lose data of course."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "41f534d4-bfcb-4cdc-9728-7a96b59a2aea",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "2ee14686-232e-4740-8be8-f31a01305485",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "### Compressors\n",
    "\n",
    "A big part of the performance of Zarr is due to its support for compression of individual chunks. Zarr by default supports 20 different compression codecs. These live in a separate package called [numcodecs](https://numcodecs.readthedocs.io/en/stable/). The default compressor is the [Blosc](https://www.blosc.org/) meta-compressor. It's easy to add a new compressor or filter.\n",
    "\n",
    "For the sake of time, we've decided to skip going into detail on compressors. You can read the [Zarr docs](https://zarr.readthedocs.io/en/stable/tutorial.html#compressors) for more information. The default compressor usually works well for most applications."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "994023da-d074-40cc-8e95-cec3d77711ef",
   "metadata": {},
   "source": [
    "### Groups\n",
    "\n",
    "To keep many arrays together, we can organize them in groups."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "157db8e7-6c04-4b6d-a89f-bb77e7a40823",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ca81301-fd0b-46e0-b28e-631b70facb3b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "686d6e0a-9145-41fb-83df-b447ec34adf1",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Zarr in the Cloud"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c061e395-63e9-4191-bd08-3bde9206a609",
   "metadata": {},
   "source": [
    "### Writing to and Reading from Cloud Object Storage\n",
    "\n",
    "Zarr can store data in any storage system that can be represented as a key-value store. Here are some examples\n",
    "\n",
    "- A directory on your filesystem\n",
    "- A ZipFile\n",
    "- A Redis database\n",
    "- A cloud object store (e.g. S3, GCS, Azure Blob Storage)\n",
    "\n",
    "In the cell below, we will access an S3 bucket with read-write credentials. These credentials will be disabled after the tutorial.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "570fa7fd-7476-4e99-8d44-ea368f0a09ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "import uuid\n",
    "storage_kwargs = {\n",
    "    \"key\": \"AKIATRRGXBOIEPMO524C\",\n",
    "    \"secret\": \"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\"\n",
    "}\n",
    "my_folder = f\"s3://pangeo-ogc-demo/{uuid.uuid4().hex}\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4836199-bd71-46ee-8f6f-0de5d7079081",
   "metadata": {},
   "source": [
    "Now we create a store in that location and store a group in it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e2dcd11-6485-4a36-ade7-5434e934baa0",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a9fc7edc-c4f1-418a-a933-f72132af1ca7",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5142e37b-ddf4-4934-8f30-8ec36bddc79b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "4af53357-2cda-4899-b107-0121fcdb3e75",
   "metadata": {},
   "source": [
    "### Consolidating Metadata\n",
    "\n",
    "Listing directories can sometimes be slow (or impossible) on certain storage devices. Zarr offers the ability to consilidate the metadata for an entire group into a single object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99e8c806-6707-4528-842f-686ab881460e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b47f791-d816-4dc3-b449-5397f876d044",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "92886b92-384d-4473-857b-935ec58a4fb9",
   "metadata": {},
   "source": [
    "## Zarr + Xarray (+ Dask)\n",
    "\n",
    "Confession: I almost never use the `zarr` library directly. I nearly always read and write Zarr via [Xarray](https://xarray.dev/) because I like Xarray's data model. (Xarray's data model is the NetCDF data model).\n",
    "\n",
    "> Xarray is an open source project and Python package that introduces labels in the form of dimensions, coordinates, and attributes on top of raw NumPy-like arrays, which allows for more intuitive, more concise, and less error-prone user experience.\n",
    "> \n",
    "> Xarray includes a large and growing library of domain-agnostic functions for advanced analytics and visualization with these data structures.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30d8529e-3718-4d80-958a-8865d6bf62c4",
   "metadata": {},
   "source": [
    "### Quick Review of Xarray"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3116507f-47b9-4446-9248-dad343bc33a9",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d9e1b31-9754-49f7-9664-271ca0ea7269",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c12a0149-a96a-461f-8aa2-1b527e15647c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "816ada86-69b1-4f4c-9008-696a1e4d64f7",
   "metadata": {},
   "source": [
    "### Writing Zarr from Xarray\n",
    "\n",
    "First we chunk the dataset. This accomplishes two things.\n",
    "- Allows parallel processing using [Dask](https://dask.org/) (Not necessary for this small-data example but very useful for big data)\n",
    "- Automatically maps Dask chunks to Zarr chunks when writing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "64d74e98-fa7b-4eb3-b192-088129ce4b36",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5cf49c0c-bb88-43e1-b6eb-506edff65803",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27fe28fd-9017-4c44-87f4-8efe37576695",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "3347240a-d47a-416c-b7d1-31c9a548c98c",
   "metadata": {},
   "source": [
    "### Reading from S3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b2a2890-094d-419f-8d7f-64b1e5a783c3",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ef563e7-f590-41a1-a358-88770b5c16cb",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "c2681cb5-f0c9-4757-a1f3-834a5e9172a5",
   "metadata": {
    "tags": []
   },
   "source": [
    "Here is some syntactic sugar to automatically map the Zarr chunks to Dask chunks when reading. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a748a25c-2083-412d-b89f-1f530650bece",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "14a77478-6309-44fd-b748-1479100c80d9",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50c1940d-6ff4-47a6-8be3-de9e243f133d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "3d71d4b4-0568-4429-8325-bbff9ba222e1",
   "metadata": {},
   "source": [
    "### Let's Look under the hood"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2decae2e-321f-403b-bf8c-9ed16ad5c107",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4c4521e5-e051-490a-a5b3-9a06b52c1cc1",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee76ef8a-f01f-4cf2-8141-dc69bbfd6521",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "d839de4e-5d65-40a3-a5f0-588d7bdadae1",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Example Datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35667caf-7a69-4809-9b6f-48dacfc912cb",
   "metadata": {},
   "source": [
    "### Work with Existing CMIP6 cloud data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92629373-3b31-4605-8ae7-1b8dfee41fbb",
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_url = \"https://cmip6-pds.s3-us-west-2.amazonaws.com/pangeo-cmip6.csv\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71b35739-a670-4eab-a536-d5e125a56946",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7cd61f52-22c0-401a-a130-e7968a7e6720",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a4c0cd1-4f9e-4206-b3f9-c50a2bfdb3a2",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f30d2a8d-3d8d-4f1a-9dff-04233877c42b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5d58453-797d-4299-b544-e3621cd91aec",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8abcbff4-6588-4ec8-a4f2-4f03ad6c41c9",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "643230ff-c7df-49ce-b3cd-b7e340aa09c4",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e414991-39b6-465c-a9dc-614b37019c3c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "c2c348f8-a483-4b20-ba12-1ea1c83c4c1f",
   "metadata": {},
   "source": [
    "### MUR SST\n",
    "\n",
    "https://registry.opendata.aws/mur/\n",
    "\n",
    "> A global, gap-free, gridded, daily 1 km Sea Surface Temperature (SST) dataset created by merging multiple Level-2 satellite SST datasets. Those input datasets include the NASA Advanced Microwave Scanning Radiometer-EOS (AMSR-E), the JAXA Advanced Microwave Scanning Radiometer 2 (AMSR-2) on GCOM-W1, the Moderate Resolution Imaging Spectroradiometers (MODIS) on the NASA Aqua and Terra platforms, the US Navy microwave WindSat radiometer, the Advanced Very High Resolution Radiometer (AVHRR) on several NOAA satellites, and in situ SST observations from the NOAA iQuam project. Data are available from 2002 to present in Zarr format. The original source of the MUR data is the NASA JPL Physical Oceanography DAAC."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8be45c16-96fd-4dfd-ae73-e16fa31ca0cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "mur_url = 's3://mur-sst/zarr/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75dac07b-6245-499e-96eb-3a4067b44838",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}