JP-3753: Refactor extract1d

[melanieclarke]

Resolves JP-3753

Significant refactor for extract1d, to improve code organization, maintainability, and efficiency, as well as set up for PSF-based extraction, to be added in the future.

Tasks

• request a review from someone specific, to avoid making the maintainers review every PR
• add a build milestone, i.e. Build 11.3 (use the latest build if not sure)
• Does this PR change user-facing code / API? (if not, label with no-changelog-entry-needed)
  • write news fragment(s) in changes/: echo "changed something" > changes/<PR#>.<changetype>.rst (see below for change types)
  • update or add relevant tests
  • update relevant docstrings and / or docs/ page
  • start a regression test and include a link to the running job (click here for instructions)
    • Do truth files need to be updated ("okified")?
      • after the reviewer has approved these changes, run okify_regtests to update the truth files
• if a JIRA ticket exists, make sure it is resolved properly

news fragment change types...

• changes/<PR#>.general.rst: infrastructure or miscellaneous change
• changes/<PR#>.docs.rst
• changes/<PR#>.stpipe.rst
• changes/<PR#>.datamodels.rst
• changes/<PR#>.scripts.rst
• changes/<PR#>.fits_generator.rst
• changes/<PR#>.set_telescope_pointing.rst
• changes/<PR#>.pipeline.rst

stage 1

• changes/<PR#>.group_scale.rst
• changes/<PR#>.dq_init.rst
• changes/<PR#>.emicorr.rst
• changes/<PR#>.saturation.rst
• changes/<PR#>.ipc.rst
• changes/<PR#>.firstframe.rst
• changes/<PR#>.lastframe.rst
• changes/<PR#>.reset.rst
• changes/<PR#>.superbias.rst
• changes/<PR#>.refpix.rst
• changes/<PR#>.linearity.rst
• changes/<PR#>.rscd.rst
• changes/<PR#>.persistence.rst
• changes/<PR#>.dark_current.rst
• changes/<PR#>.charge_migration.rst
• changes/<PR#>.jump.rst
• changes/<PR#>.clean_flicker_noise.rst
• changes/<PR#>.ramp_fitting.rst
• changes/<PR#>.gain_scale.rst

stage 2

• changes/<PR#>.assign_wcs.rst
• changes/<PR#>.badpix_selfcal.rst
• changes/<PR#>.msaflagopen.rst
• changes/<PR#>.nsclean.rst
• changes/<PR#>.imprint.rst
• changes/<PR#>.background.rst
• changes/<PR#>.extract_2d.rst
• changes/<PR#>.master_background.rst
• changes/<PR#>.wavecorr.rst
• changes/<PR#>.srctype.rst
• changes/<PR#>.straylight.rst
• changes/<PR#>.wfss_contam.rst
• changes/<PR#>.flatfield.rst
• changes/<PR#>.fringe.rst
• changes/<PR#>.pathloss.rst
• changes/<PR#>.barshadow.rst
• changes/<PR#>.photom.rst
• changes/<PR#>.pixel_replace.rst
• changes/<PR#>.resample_spec.rst
• changes/<PR#>.residual_fringe.rst
• changes/<PR#>.cube_build.rst
• changes/<PR#>.extract_1d.rst
• changes/<PR#>.resample.rst

stage 3

• changes/<PR#>.assign_mtwcs.rst
• changes/<PR#>.mrs_imatch.rst
• changes/<PR#>.tweakreg.rst
• changes/<PR#>.skymatch.rst
• changes/<PR#>.exp_to_source.rst
• changes/<PR#>.outlier_detection.rst
• changes/<PR#>.tso_photometry.rst
• changes/<PR#>.stack_refs.rst
• changes/<PR#>.align_refs.rst
• changes/<PR#>.klip.rst
• changes/<PR#>.spectral_leak.rst
• changes/<PR#>.source_catalog.rst
• changes/<PR#>.combine_1d.rst
• changes/<PR#>.ami.rst

other

• changes/<PR#>.wfs_combine.rst
• changes/<PR#>.white_light.rst
• changes/<PR#>.cube_skymatch.rst
• changes/<PR#>.engdb_tools.rst
• changes/<PR#>.guider_cds.rst

[codecov]

Codecov Report

Attention: Patch coverage is 88.47458% with 34 lines in your changes missing coverage. Please review.

Project coverage is 66.69%. Comparing base (a9ab10a) to head (bb7b338).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
jwst/extract_1d/extract_1d_step.py	72.17%	32 Missing ⚠️
jwst/extract_1d/ifu.py	94.11%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8961      +/-   ##
==========================================
+ Coverage   64.54%   66.69%   +2.15%     
==========================================
  Files         375      375              
  Lines       38737    37786     -951     
==========================================
+ Hits        25002    25201     +199     
+ Misses      13735    12585    -1150     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[melanieclarke]

Initial regression test run here:
https://github.com/spacetelescope/RegressionTests/actions/runs/11846882332
Total test time is about half an hour shorter than the last scheduled run on main.

There are differences in every x1d and x1dints product, but they are all minor:

• DQ array is now updated to set DO_NOT_USE where output flux is NaN. The previous version was not setting DQ in a useful way.
• Flux and background errors are different for apertures that include partial pixels. This is because the variance propagation in the current version is incorrect: variances in partial pixels are currently weighted by the pixel fraction, like the science data is, but they should be weighted by pixel fraction squared. The weights are fixed in this version.
• In x1dints files, extraction start/stop values are now added to the headers for spectra from all integrations, not just the first integration.
• Left and right extraction limit values are now directly set from the aperture edges, where they first go to zero, instead of from the input extraction parameters. This makes a one pixel difference to the MIRI LRS slitless EXTRYSTR/EXTRYSTP values (test_miri_lrs_slitless_tso_spec2[x1dints]).
• Approximate extraction start/stop values are added to the headers for spectra extracted with polynomial src_coeff parameters. Before, there was no information about the extraction location in this case (test_nirspec_bots_custom_extraction).

This small change also appears in these results, but probably shouldn't:

• Background values are usually zero if there is no background data, but they are now sometimes NaN at the edges of spectra, where they used to be zero.

I will modify the code to make sure all background values are finite. Test attempt 2 at the same link above incorporates this change.