Initial commit

.gitattributes

@@ -0,0 +1 @@
+*.qmd linguist-language=markdown

.github/workflows/quarto-publish.xxyml

@@ -0,0 +1,36 @@
+on:
+  push:
+    branches: main
+
+name: Render and Publish
+
+jobs:
+  build-deploy:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+    steps:
+      - name: Check out repo
+        uses: actions/checkout@v3
+
+      - name: Install R
+        uses: r-lib/actions/setup-r@v2
+
+      - name: Set up Quarto
+        uses: quarto-dev/quarto-actions/setup@v2
+        with:
+          # To install LaTeX to build PDF book 
+          tinytex: true 
+          # uncomment below and fill to pin a version
+          # version: 0.9.600
+
+      # add software dependencies here
+      - name: Install packages (needed for Rmd)
+        run: Rscript -e 'install.packages(c("rmarkdown", "knitr", "jsonlite", "here"))'
+
+      - name: Publish to GitHub Pages (and render)
+        uses: quarto-dev/quarto-actions/publish@v2
+        with:
+          target: gh-pages
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # this secret is always available for github actions

.gitignore

@@ -0,0 +1,15 @@
+*netrc
+*checkpoints
+/tutorials/data/*Links.txt
+*.vrt
+.Rproj.user
+.Rhistory
+.RData
+.Ruserdata
+
+/.quarto/
+/_site/
+/_freeze/
+/python-notebooks/demonstrated data/
+.DS_Store
+.DS_Store

2021-Cloud-Hackathon.Rproj

@@ -0,0 +1,13 @@
+Version: 1.0
+
+RestoreWorkspace: Default
+SaveWorkspace: Default
+AlwaysSaveHistory: Default
+
+EnableCodeIndexing: Yes
+UseSpacesForTab: Yes
+NumSpacesForTab: 2
+Encoding: UTF-8
+
+RnwWeave: Sweave
+LaTeX: pdfLaTeX

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2021 NASA Openscapes
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,33 @@
+
+## NOAA HackDays 2024: Integrating Earth Data into Science Workflows
+
+### Overview
+
+The the era of big data in the earth sciences is here and learning how to effectively use oceanographic remote-sensing data, both in the cloud and on your computer, is a core skill for modern fisheries science and management. Learning how to access cloud-based data, visualize these data, use these data in models, and use the tools of modern reproducible and collaborative science is the main goal of these hackday events. Through these events, participants will gain experience with assessing remote-sensing data in the cloud, R and RStudio, Python and Jupyter notebooks, and collaborating with Git and GitHub. 
+
+### Aims and Objectives
+
+* Learn how to discover and use oceanographic remote-sensing data for species distribution modeling and other fisheries applications
+* Familiarize participants with using remote-sensing data in R and Python with code. 
+* Obtain hands-on experience in using remote-sensing data for various science applications.
+* Learn by working together on a group project. 
+
+### What to expect
+
+* What is a hack event? See the [description of hackweeks](https://escience.washington.edu/using-data-science/hackweeks/) on the University of Washington eScience institute website:
+
+*A hackweek is a participant-driven workshop that blends data science education, community building, and project work over a short period of time (one to two weeks). The events are highly immersive and allow participants to work directly with data science professionals to co-shape projects and educational outcomes. Hackweeks often help individuals and teams engage more effectively in open and reproducible science.* - eScience Institute, University of Washington
+
+* All tutorials and examples are developed openly and will be publicly available during and following the event. Participants will strengthen their practice of open science, using open source code and collaborating on their projects with course peers.
+
+<hr>
+
+### Disclaimer
+
+This repository is a scientific product and is not official communication of the National Oceanic and Atmospheric Administration, or the United States Department of Commerce. All NOAA GitHub project content is provided on an ‘as is’ basis and the user assumes responsibility for its use. Any claims against the Department of Commerce or Department of Commerce bureaus stemming from the use of this GitHub project will be governed by all applicable Federal law. Any reference to specific commercial products, processes, or services by service mark, trademark, manufacturer, or otherwise, does not constitute or imply their endorsement, recommendation or favoring by the Department of Commerce. The Department of Commerce seal and logo, or the seal and logo of a DOC bureau, shall not be used in any manner to imply endorsement of any commercial product or activity by DOC or the United States Government.
+
+### License
+
+This content was created by U.S. Government employees as part of their official duties. This content is not subject to copyright in the United States (17 U.S.C. §105) and is in the public domain within the United States of America. Additionally, copyright is waived worldwide through the CC0 1.0 Universal public domain dedication.
+
+

_import/assets.json

@@ -0,0 +1,18 @@
+[
+    {
+        "title": "Zarr Example",
+        "preamble": "This notebook is from NASA's Zarr **EOSDIS** store notebook",
+        "source": "https://github.com/nasa/zarr-eosdis-store",
+        "url": "https://raw.githubusercontent.com/nasa/zarr-eosdis-store/main/presentation/example.ipynb",
+        "target": "zarr-eosdis-store.ipynb",
+        "process": true
+    },
+    {
+        "title": "COF Zarr Access via Reformat",
+        "preamble": "This notebook is from NASA's PO.DAAC, Access ECCO data via Harmony and the Zarr reformatter service example",
+        "source": "https://github.com/podaac/ECCO",
+        "url": "https://raw.githubusercontent.com/podaac/ECCO/main/Data_Access/cloud_harmony_zarr_reformat.ipynb",
+        "target": "cof-zarr-reformat.ipynb",
+        "process": true
+    }
+]

_import/environment.yml

@@ -0,0 +1,13 @@
+name: quarto-import
+channels:
+- conda-forge
+dependencies:
+- python=3.8
+- nbformat=5.1
+- requests=2.26
+- pytest=6.2
+- pytest-watch=4.1
+- pip
+- pip:
+  - pqdm
+  - markdown

_import/quarto_import.py

@@ -0,0 +1,74 @@
+#!/usr/bin/env python
+import argparse
+import json
+import os
+from datetime import date
+
+import requests
+import markdown
+import nbformat as nbf
+from pqdm.processes import pqdm
+
+IMPORTED_PATH = '../external'
+
+
+def import_remote(url, target):
+    if not os.path.exists(IMPORTED_PATH):
+        print('recreating local')
+        os.mkdir(IMPORTED_PATH)
+    remote_file = requests.get(url)
+    with open(f'{IMPORTED_PATH}/{target}', 'wb') as f:
+        f.write(remote_file.content)
+    return None
+
+def inject_content(content, notebook):
+    nb = nbf.read(f'{IMPORTED_PATH}/{notebook}', as_version=4)
+    preamble_cell = nbf.v4.new_markdown_cell(content)
+    # for some reason v4 says id is not part of the schema.
+    # preamble_cell.pop('id', None)
+    nb['cells'].insert(0, preamble_cell)
+    nbf.write(nb, f'{IMPORTED_PATH}/{notebook}')
+
+def create_preamble_cell(document):
+    target = document['target']
+    url = document['url']
+    source_link = f"[{document['source']}]({document['source']})"
+    import_date = date.today().strftime('%Y-%m-%d')
+    cell_content = "\n\n".join( [f"# {document['title']}",
+                                   f"imported on: **{import_date}**",
+                                   f"{markdown.markdown(document['preamble'])}",
+                                   f"> The original source for this document is {source_link}"])
+    return cell_content
+
+def process_document(document):
+    local_target = document['target']
+    url = document['url']
+    import_remote(url, local_target)
+    if document['process'] is True:
+        preamble_cell =  create_preamble_cell(document)
+        inject_content(preamble_cell, local_target)
+    print(f'Processed: {local_target}')
+
+def main(assets):
+    """
+    This module will fetch a URL, save it locally and inject some preamble
+    The currently supported formats are: .ipynb
+    """
+    for document in assets:
+        process_document(document)
+    # result = pqdm(json_input, process_document, n_jobs=2)
+    return None
+
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser(description="Ipython importer")
+    # parameters
+    parser.add_argument('-f',
+                        '--file',
+                        help = "File to parse",
+                        type = str)
+    args=parser.parse_args()
+
+    with open(args.file) as f:
+        assets = json.load(f)
+    result = main(assets)

_import/test_quarto_import.py

@@ -0,0 +1,30 @@
+import os
+import shutil
+
+import nbformat as nbf
+import pytest
+
+from quarto_import import import_remote, inject_content
+
+LOCAL_IMPORT_PATH = '../external'
+
+
+def test_quarto_import_can_import_remote():
+    remote_url = 'https://raw.githubusercontent.com/nasa-jpl/itslive-explorer/main/notebooks/itslive-notebook-rendered.ipynb'
+    import_remote(remote_url, '_test.ipynb')
+    assert os.path.exists(f'{LOCAL_IMPORT_PATH}/_test.ipynb') is True
+
+
+def test_quarto_import_can_inject_content():
+    remote_url = 'https://raw.githubusercontent.com/nasa-jpl/itslive-explorer/main/notebooks/itslive-notebook-rendered.ipynb'
+
+    notebook = '_test.ipynb'
+    content = f"""\
+    # This is a test, should be at the top
+    """
+    inject_content(content, notebook)
+    nb = nbf.read(f'{LOCAL_IMPORT_PATH}/_test.ipynb', as_version=4)
+    assert nb['cells'][0]['cell_type'] == 'markdown'
+    assert content in nb['cells'][0]['source']
+    # Needs to be refactored into a local clean
+    import_remote(remote_url, notebook)