NIRISS SOSS Background Correction Implementation

[stscijgbot-jp]

Issue JP-3825 was created on JIRA by Aarynn Carter:

Context: NIRISS SOSS observations exhibit a spatially varying background that must be subtracted prior to spectral extraction to maximize the achievable precision. Thus far, a single "template" background measured from on-sky observations has been used for this, although my understanding is that this correction is not actually applied in the pipeline. In reality, we have found that the background has significant variability as a function of sky position and time, and a single template background is not sufficient for an optimal correction. As such the NIRISS SOSS team has observed and collated a small library of background observations to be used for background subtraction purposes. I have attached a gif of subarray rate images to give you a visual idea of what the background looks like, note the sharp increase at x~700 and it's gradual dimming to later columns. 

Implementation: We would like to implement a background correction for NIRISS SOSS observations, using the library of backgrounds we have produced. However, we think this will require a somewhat careful implementation due to interactions with the column-wise 1/f noise. Ideally, we would like to retain the capability for a group-level 1/f correction without negatively impacting the background correction, and if possible within the framework of the likelihood-based jump detection and ramp fitting routines. My expectation is that this will break the standard flow of the pipeline, but my current best guess for how this would work is:

1. Process the images to a rate level image.

2. Fit and determine a best matching background model.

3. Return to the group-level images and produce a set of background subtracted images. 

4. Use these images to estimate a group-level 1/f noise correction.

5. Apply this correction to the original (not background subtracted) group-level images.

6. Process the 1/f corrected images to the integration level. 

7. Potentially repeat the background model fitting and subtract the background from the integration level images. 

Throughout this process, I would also like to retain some flexibility for users in being able to provide their own background model, instead of relying entirely on our library. This will also future-proof things for us in the event that we develop new background fitting methods. 

As you can see things are quite intertwined, so I am interested in having a broader discussion about how best to move forward. Nominally, it would be great to implement this in what I'm assuming will be the 11.4 build. But, given the complexity, we thought it was worth giving you as much of a heads up as possible so that we can figure out a best path forward together. 

Note that this connects heavily with #⁠277 

[stscijgbot-jp]

Comment by Aarynn Carter on JIRA:

This was discussed further in the weekly SOSS meeting and we would also be open to considering a solution where the correction is applied at Stage 2, as it may be possible for us to measure the 1/f noise more straightforwardly by subtracting a rolling median of successive integrations prior to ramp fitting. What isn't clear to us is how that interacts with the 1/f and ramp fitting routines, and whether it is actually preferable over the method outlined above. In any event, it would still require computing a best matching background model on the highest SNR, integration-stacked, data.