PCMDI Utils

PCMDI Utils aims to host reuseable functions across the PCMDI project in one place.

The target audience was originally developers from the PCMDI, E3SM Project, and LLNL Climate Program, but it can be used by anyone in the open-source community!

The repository is initiated by using the Python Template Repo generated by @tomvothecoder

CURRENTLY UNDER DEVELOPMENT!!

Features

• Python library boilerplate code using standard practices
• Unit testing framework using pytest
• Quality assurance tooling (code formatting and linting)
• Sphinx documentation with Read the Docs Sphinx Theme
• GitHub Actions to run CI/CD workflows for unit testing and quality assurance
• GitHub Issue and Pull Request templates for organized ticket tracking
• Anaconda environments defined in .yml files for determinism
• Licensed with Apache-2.0 and .gitignore tailored to Python projects

Getting Started

CURRENTLY UNDER CONSTRUCTION

Resources for developers

1. Clone this repository

Follow the steps below to get this repository cloned to your local environment

git clone [email protected]:PCMDI/pcmdi_utils.git

2. Set up the Anaconda Development Environment

Follow the steps below to get the Anaconda development environment set up for your library. We recommend using mamba instead of conda because it is significantly faster and more reliable.

1. Download and Install Anaconda (Mac OS & Linux)

   curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-$(uname)-$(uname -m).sh"
   bash Mambaforge-$(uname)-$(uname -m).sh

2. Update /conda-env/*.yml by follow each file's #TODO comments (don't worry, it's straightforward)

3. Create and activate the Conda development environment

   mamba env create -f conda-env/dev.yml
   mamba activate template-dev

3. Set up the Documentation

The library's documentation is setup with Sphinx and Read the Docs Sphinx Theme.

You just need to do a few things to get up and running:

1. Update README.rst, AUTHORS.rst, HISTORY.rst, and /docs/index.rst as needed
2. Decide how to deploy the documentation. This step is left up to you based on your needs. Follow the instructions in the provided links.
   • Option 1: Read the Docs
   • Option 2: GitHub Pages (via GitHub Actions)

To build the documentation locally (useful for reviewing):

1. Activate the conda developer environment

   mamba activate template-dev

2. Build documentation

   • Option 1: make docs from the root of repo (also cleans up and opens docs in your browser)
   • Option 2: cd docs && make html

4. Set up Quality Assurance Tools

This repository includes quality assurance (QA) tools for code formatting (black, isort), linting (flake8), unit testing (pytest), and optional static type checking (mypy). These tools ensure that you can easily catch issues and follow good Python practices without sacrificing energy on them. These tools are integrated in the pre-commit package as "hooks" that automatically run when committing changes to files.

To run these QA tools through pre-commit:

1. Activate the conda development environment

   mamba activate template-dev

2. Install pre-commit hooks into the Git repository

   $ pre-commit install
   pre-commit installed at .git/hooks/pre-commit

   • pre-commit will now run automatically on git commit!

3. (optional) Run against all files

   • it's usually a good idea to run the hooks against all of the files when adding new hooks (usually pre-commit will only run on the changed files during git hooks)

   $ pre-commit run --all-files
   [INFO] Initializing environment for https://github.com/pre-commit/pre-commit-hooks.
   [INFO] Initializing environment for https://github.com/psf/black.
   [INFO] Installing environment for https://github.com/pre-commit/pre-commit-hooks.
   [INFO] Once installed this environment will be reused.
   [INFO] This may take a few minutes...
   [INFO] Installing environment for https://github.com/psf/black.
   [INFO] Once installed this environment will be reused.
   [INFO] This may take a few minutes...
   Check Yaml...............................................................Passed
   Fix End of Files.........................................................Passed
   Trim Trailing Whitespace.................................................Failed
   - hook id: trailing-whitespace
   - exit code: 1
   
   Files were modified by this hook. Additional output:
   
   Fixing sample.py
   
   black....................................................................Passed

Useful commands:

• Override pre-commit checks: git commit -m <MESSAGE> --no-verify
• Run individual hook: pre-commit run --all-files <black|flake8|isort>

Information on QA tools:

• pre-commit - "Git hook scripts are useful for identifying simple issues before submission to code review."
• black - "The uncompromising code formatter" that "will save time and mental energy for more important matters".
• isort - isort is a Python utility / library to sort imports alphabetically, and automatically separated into sections and by type.
• flake8 - A Python linter that checks Python code for style and syntax errors, and for enforcing a style guide with PEP (Python Enhancement Proposals).
• mypy (enable in pre-commit-config.yaml) - A static type checker for Python. "Type checkers help ensure that you’re using variables and functions in your code correctly. With mypy, add type hints (PEP 484) to your Python programs, and mypy will warn you when you use those types incorrectly."

5. Decide How to Distribute Software on Anaconda

There are several ways of distributing software on Anaconda including:

1. Using conda-forge (recommended)
2. Managing your own Anaconda channel and uploading packages there

I recommend using conda-forge. "conda-forge is a GitHub organization containing repositories of conda recipes. Thanks to some awesome continuous integration providers (AppVeyor, Azure Pipelines, CircleCI and TravisCI), each repository, also known as a feedstock, automatically builds its own recipe in a clean and repeatable way on Windows, Linux and OSX.""

Helpful Knowledge

Reminder to Routinely Maintain the Software Infrastructure

It is generally good practice to maintain your software infrastructure. Here is a routine software maintenance checklist. I recommend doing these before every new software version release.

1. Maintain conda environment dependencies in .yml files
2. Update pre-commit hooks in .pre-commit-config.yml

How GitHub Actions is Used for CI/CD

This repository uses GitHub Actions to run the CI/CD build workflow. This workflow is automatically triggered with commits on pull requests to main and merging pull requests to main.

Jobs include:

1. Run pre-commit for formatting, linting, and type checking
2. Build conda CI/CD environment with different Python versions, build and install the package, and run unit test suite with pytest

Here's an example of GitHub Actions in "action": https://github.com/tomvothecoder/python-template-repo/actions

To save time and resources on build cycles, GitHub Actions has been configured to automatically skip jobs instead of re-executing based on the files that are committed. For example, if docs are committed then the unit tests will not run.

Additional Configuration

• Feel free to modify the configuration for QA tools in pyproject.toml and setup.cfg
• You can also remove some or all the QA tools if you want (not recommended though). Just make sure to delete them from your Anaconda .yml files and remove their entries in pre-commit-config.yaml and their configs in pyproject.toml and/or setup.cfg.
• mypy is disabled by default with pre-commit. Enable mypy by uncommenting the lines related lines in pre-commit-config.yaml.