JP-3799: First frame bright

[karllark]

Resolves JP-3799

This PR modifies the firstframe step to optionally not flag the 1st group (aka frame) as DO_NOT_USE for pixels that have ramps that saturation between the 2nd and 3rd groups. In the case of a pixel having a large signal, it has been found that the slope estimated from the 2nd-1st groups and 3rd-2nd groups is equivalent within 1% indicating that the firstframe effect is small compared to the signal in this specific case. This means that for pixels seeing very bright signal (i.e., saturating between the 2nd and 3rd groups), that the slope can be measured from the 1st and 2nd groups. This provides a slope measurement where previously there was none (i.e., a NaN value).

Tasks

• request a review from someone specific, to avoid making the maintainers review every PR
• add a build milestone, i.e. Build 11.3 (use the latest build if not sure)
• Does this PR change user-facing code / API? (if not, label with no-changelog-entry-needed)
  • write news fragment(s) in changes/: echo "changed something" > changes/<PR#>.<changetype>.rst (see below for change types)
  • update or add relevant tests
  • update relevant docstrings and / or docs/ page
  • start a regression test and include a link to the running job (click here for instructions)
    • Do truth files need to be updated ("okified")?
      • after the reviewer has approved these changes, run okify_regtests to update the truth files
• if a JIRA ticket exists, make sure it is resolved properly

news fragment change types...

• changes/<PR#>.general.rst: infrastructure or miscellaneous change
• changes/<PR#>.docs.rst
• changes/<PR#>.stpipe.rst
• changes/<PR#>.datamodels.rst
• changes/<PR#>.scripts.rst
• changes/<PR#>.fits_generator.rst
• changes/<PR#>.set_telescope_pointing.rst
• changes/<PR#>.pipeline.rst

stage 1

• changes/<PR#>.group_scale.rst
• changes/<PR#>.dq_init.rst
• changes/<PR#>.emicorr.rst
• changes/<PR#>.saturation.rst
• changes/<PR#>.ipc.rst
• changes/<PR#>.firstframe.rst
• changes/<PR#>.lastframe.rst
• changes/<PR#>.reset.rst
• changes/<PR#>.superbias.rst
• changes/<PR#>.refpix.rst
• changes/<PR#>.linearity.rst
• changes/<PR#>.rscd.rst
• changes/<PR#>.persistence.rst
• changes/<PR#>.dark_current.rst
• changes/<PR#>.charge_migration.rst
• changes/<PR#>.jump.rst
• changes/<PR#>.clean_flicker_noise.rst
• changes/<PR#>.ramp_fitting.rst
• changes/<PR#>.gain_scale.rst

stage 2

• changes/<PR#>.assign_wcs.rst
• changes/<PR#>.badpix_selfcal.rst
• changes/<PR#>.msaflagopen.rst
• changes/<PR#>.nsclean.rst
• changes/<PR#>.imprint.rst
• changes/<PR#>.background.rst
• changes/<PR#>.extract_2d.rst
• changes/<PR#>.master_background.rst
• changes/<PR#>.wavecorr.rst
• changes/<PR#>.srctype.rst
• changes/<PR#>.straylight.rst
• changes/<PR#>.wfss_contam.rst
• changes/<PR#>.flatfield.rst
• changes/<PR#>.fringe.rst
• changes/<PR#>.pathloss.rst
• changes/<PR#>.barshadow.rst
• changes/<PR#>.photom.rst
• changes/<PR#>.pixel_replace.rst
• changes/<PR#>.resample_spec.rst
• changes/<PR#>.residual_fringe.rst
• changes/<PR#>.cube_build.rst
• changes/<PR#>.extract_1d.rst
• changes/<PR#>.resample.rst

stage 3

• changes/<PR#>.assign_mtwcs.rst
• changes/<PR#>.mrs_imatch.rst
• changes/<PR#>.tweakreg.rst
• changes/<PR#>.skymatch.rst
• changes/<PR#>.exp_to_source.rst
• changes/<PR#>.outlier_detection.rst
• changes/<PR#>.tso_photometry.rst
• changes/<PR#>.stack_refs.rst
• changes/<PR#>.align_refs.rst
• changes/<PR#>.klip.rst
• changes/<PR#>.spectral_leak.rst
• changes/<PR#>.source_catalog.rst
• changes/<PR#>.combine_1d.rst
• changes/<PR#>.ami.rst

other

• changes/<PR#>.wfs_combine.rst
• changes/<PR#>.white_light.rst
• changes/<PR#>.cube_skymatch.rst
• changes/<PR#>.engdb_tools.rst
• changes/<PR#>.guider_cds.rst

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 64.53%. Comparing base (5a2d0cd) to head (3401d49).
Report is 19 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #8952   +/-   ##
=======================================
  Coverage   64.52%   64.53%           
=======================================
  Files         375      375           
  Lines       38739    38745    +6     
=======================================
+ Hits        24997    25003    +6     
  Misses      13742    13742           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests

[jemorrison]

Code looks good and the test added checks that it is working.

[melanieclarke]

Looks good! One quick suggestion for the docs for the new argument.

I'll start regression tests.

[melanieclarke]

Regression tests running here:
https://github.com/spacetelescope/RegressionTests/actions/runs/11898230791

All tests pass.

[drlaw1558]

It might be useful to expand this to unflag group 1 in the cases where saturation occurs in group 2 as well, not just between groups 2 and 3. This would allow for potential future support of single-group data (e.g., using the suppress_one_group=False approach) without having to disable the first-frame step entirely.

[karllark]

It might be useful to expand this to unflag group 1 in the cases where saturation occurs in group 2 as well, not just between groups 2 and 3. This would allow for potential future support of single-group data (e.g., using the suppress_one_group=False approach) without having to disable the first-frame step entirely.

I agree this is a good idea. I will update the PR to make this change. It is a straightforward change.

[melanieclarke]

Thanks for the update - I'll re-run regression tests.

It looks like some of the docs are still referring to saturating between 2 and 3 - should those be updated, since group 2 can also now be saturated?

[karllark]

It looks like some of the docs are still referring to saturating between 2 and 3 - should those be updated, since group 2 can also now be saturated?

Good catch. I've (hopefully) updated things in all the right places.

[melanieclarke]

I found one last place that was referring to saturating between group 2 and 3 - suggestion below.

I reran regression tests at the same link above. There are some unrelated failures, but nothing related to this PR.

I'll approve now from my perspective. @drlaw1558 - are you happy with this one now?

[drlaw1558]

LG2M

[karllark]

I've fixed the last doc issue and pushed the commit. Seems to be taking sometime for github to process this update. Just a FYI.

[melanieclarke]

@karllark - I still don't see the update to the docs. Can you check that you pushed all the changes up?

[karllark]

I did. At the top of this PR you can see a wheel going around saying there has been an update, just hasn't been ingested somehow yet.

[karllark]

And on my fork/branch the commit is present.

[melanieclarke]

Okay, thanks. I'll close and reopen and see if that triggers an update.

[melanieclarke]

All good now! I will merge.