Add support for native ERA5 data in GRIB format

[schlunma]

Description

This PR allows ESMValCore to process native ERA5 data in GRIB format, which is for example available on Levante in the /pool/data/ERA5 directory.

Reading the data

The following settings are necessary in the user configuration file:

rootpath:
  ...
  native6:
    /pool/data/ERA5: DKRZ-ERA5-GRIB
  ...

I added an extra facets file which includes reasonable default for all supported variables. You can check it out here.

Thus, reading this data is as easy as

datasets:
  - {project: native6, dataset: ERA5, timerange: '2000/2001', short_name: tas, mip: Amon}
  - {project: native6, dataset: ERA5, timerange: '2000/2001', short_name: cl, mip: Amon, tres: 1H, frequency: 1hr}
  - {project: native6, dataset: ERA5, timerange: '2000/2001', short_name: ta, mip: Amon, type: fc, typeid: '12'}

Regridding

Native ERA5 data in GRIB format is on a reduced Gaussian grid (i.e., an unstructured grid). Thus, in 99% of the use cases, it is necessary to regrid this data, especially since no cell areas are available for the data (thus, we cannot even calculate global/regional statistics over the native data). This is done automatically by the CMORizer (as recommended by the ECMWF), but can be disabled in the recipe:

datasets:
  - {project: native6, dataset: ERA5, timerange: '2000/2001', short_name: tas, mip: Amon, automatic_regrid: false}

This PR depends on the following other PRs:

• replace deprecated eccodes-python PyPI package with new eccodes in core requirements SciTools/iris-grib#357
• Refactor regridding #2231
• Cache regridding weights if possible #2344
• Added unstructured linear regridding #2350
• Make DRS path templates configurable per rootpath #1894 (optional, but very helpful)

---

Closes #1991
Closes ESMValGroup/ESMValTool#3238

Link to documentation: https://esmvaltool--2178.org.readthedocs.build/projects/ESMValCore/en/2178/quickstart/find_data.html#supported-native-reanalysis-observational-datasets

---

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🧪 The new functionality is relevant and scientifically sound
• 🛠 This pull request has a descriptive title and labels
• 🛠 Code is written according to the code quality guidelines
• 🧪 and 🛠 Documentation is available
• 🛠 Unit tests have been added
• 🛠 Changes are backward compatible
• 🛠 Any changed dependencies have been added or removed correctly
• 🛠 The list of authors is up to date
• 🛠 All checks below this pull request were successful

---

To help with the number pull requests:

• 🙏 We kindly ask you to review two other open pull requests in this repository

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.85%. Comparing base (a328578) to head (1da3d12).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2178      +/-   ##
==========================================
+ Coverage   94.66%   94.85%   +0.18%     
==========================================
  Files         251      251              
  Lines       14287    14371      +84     
==========================================
+ Hits        13525    13631     +106     
+ Misses        762      740      -22     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[schlunma]

This is ready from my side, but there's two issues that need to be resolved before I mark this ready for review:

• Cleaned and extended function that extracts datetimes from paths #2181
• For some reason I can't get iris-grib to run on CircleCI; locally it works well. This is probably closely related to the problem reported by @remi-kazeroni here.

I tested this thoroughly with the following recipe: recipe_000.yml.txt

An example run is available on Levante here: /home/b/b309141/scratch/esmvaltool_output/recipe_000_20230825_080240

Note that with the default dask scheduler, this recipe ran into a timeout after 8 hours with 67/76 tasks finished. With the following dask configuration, I could run the same recipe on the same node (regular Levante compute node with 256 GiB of memory) in 5:27 min (!!) 🚀.

cluster:
  type: distributed.LocalCluster
  n_workers: 32
  threads_per_worker: 4
  memory_limit: 8 GiB

@ESMValGroup/technical-lead-development-team