Parameter files

Jwst 1.2 implements a significant change to how JWST pipelines are configured and operate. This change should be transparent to users in the current release.

Originally, how the pipelines operated was determined by a set of configuration (CFG) files that were delivered as part of the JWST package. These configuration files were retrieved using the "collect_pipeline_cfgs" command. The configuration files were used to run each of the different pipelines using the "strun" command. For example

$ collect_pipeline_cfgs ./

$ strun calwebb_spec2.cfg an_exposure_file.fits

The issue with this approach is that changes to the configuration files require a code release.

With the latest jwst release pipeline configurations no longer depend on package-delivered configuration files. Instead, all default configuration relies on settings in the pipeline code itself, using CRDS-retrieved parameter reference files to modify any parameters that are data-dependent. There is no longer any need to run "collect_pipeline_cfgs" and specify a configuration file for the "strun" command. One only needs to specify a simplified pipeline name. In most cases, this simple name, or alias, is the same as the name of the old configuration file, but without the suffix ".cfg".

Taking the example above, to get the same operation, the single command would become:

$ strun calwebb_spec2 an_exposure_file.fits

The JWST documentation has been updated to account for this change in usage. To get familiarized, it is best to start with the introduction:

https://jwst-pipeline.readthedocs.io/en/latest/jwst/introduction.html

A list of the available pipeline aliases can be found in the Pipeline Stages section

https://jwst-pipeline.readthedocs.io/en/latest/jwst/pipeline/main.html#pipelines

An added benefit to removing the dependency on package-delivered configuration files is that users, under normal circumstances, no longer need to be concerned with configuration files and whether they are up-to-date. One only needs to install the JWST package and start using the pipelines out-of-the-box.

Does this mean that everyone has to immediately change their behavior and code if using the default configuration files? Short answer is “No”. If one wishes to continue using the package-delivered configuration files from "collect_pipeline_cfgs", one may do so. However, these configuration files no longer contain any parameter settings; only the class name of the pipeline to be run. This allows the code-plus-CRDS-retrieved parameter reference files to determine operation.

Since the configuration settings have simply been moved to CRDS, the results one obtains should not change. If a change in behavior is noted, please report the issue to the Help Desk.

In the meantime, please consider deprecating the use of "collect_pipeline_cfgs" and the .cfg files in favor of simply specifying pipeline aliases, as the documentation now describes.

For users that use their own, custom configuration files, there is no change to functionality. However, there are changes to both how these files are documented and their format.

Concerning documentation, there is a change of terminology. No longer are these files referred to as “configuration files”, but are called “parameter files” or “parameter reference files” when retrieved from CRDS.

In order to simplify integration with CRDS, the format of parameter files have changed from the “cfg”, init-like format, to the ASDF format. All parameter files in CRDS are in this format. Similarly, the tools provided by the JWST package to create parameter files will create them in ASDF. “cfg”-formatted files are still supported, but it is strongly suggested that users change to using the ASDF form. For more information, please to refer to the documentation:

https://jwst-pipeline.readthedocs.io/en/latest/jwst/introduction.html#parameter-files

As always, if anyone finds any discrepancies or other issues with the documentation, or actual operation of the pipelines, please contact the Help Desk.

JWST Development