JWST Build 7.6

Release Notes

The JWST calibration pipeline (v0.17.1) has been made available on PyPi and Github. A high-level list of updates in this build is given below.

Association generator rules have been updated to:

• Fix incorrect product names for level-3 IFU products

Updates to Level-2a processing for all exposure modes include:

• Enabled creation of the "rateints" product for all exposures, even when NINTS=1

Updates to Level-2b processing for all exposure modes include:

• Enabled application of the "filteroffset" correction for NIRCam and NIRISS imaging modes in the assign_wcs step
• The flat-field step has been updated to set DQ='DO_NOT_USE' for reference file pixels that have values of NaN or zero, as well as reference file pixels flagged as 'NO_FLAT'.

Updates to Level-2b spectroscopic (calwebb_spec2) processing include:

• The ability to perform master background subtraction for NIRSpec MOS exposures
• The ability to process NIRSpec internal lamp exposures
• Improved calculation of bounding boxes for grism images in the assign_wcs step
• Updated the assign_wcs step to allow for user-settable extraction heights in grism (WFSS) images
• The ability to apply minimal processing of MIRI MRS TSO exposures
• Changed the default weighting in the cube_build step to "emsm"
• The ability to perform aperture corrections for point source extracted spectra in the extract_1d step
• Bug fixes in flat-fielding for NIRSpec fixed-slit exposures

Updates to the Level-3 imaging (calwebb_image3) processing include:

• The outlier_detection step has been fixed so that it does not subtract background from the science data

Updates to the Level-3 spectroscopic (calwebb_spec3) processing include:

• Implemented corrections to master background subtraction for NIRSpec IFU exposures
• Updated the master_background step to allow for saving of the 2D master background data

Updates to the Level-3 Coronagraphic (calwebb_coron3) pipeline include:

• Implemented bad pixel replacement in the align_refs step
  Updates to the Level-3 TSO (calwebb_tso3) pipeline include:

• Fixed a bug in the computation of integration time stamps

Complete change log:

associations

• Add product name override to the IFUGratingBkg class, to prevent the default
  "clear" suffix showing up in NIRSpec IFU product names. [#5326]

barshadow

• Implement using a user-supplied correction which overrides all references. [#5302]

• Implement applying the inverse operation. [#5302]

blendmeta

• Do not close files that were not opened by blendmodels [#5299]

cube_build

• If every wavelength plane of the IFU cube contains 0 data, cube_build is skipped [#5294]

• Remove "clear" suffix from MIRI MRS product name templates [#5326]

flat_field

• Update how the flat field reference dq mask is used for NIRSpec MOS data [#5284]

• Implement providing a user-supplied flat field which overrides all references. [#5302]

• Implement applying the inverse operation. [#5302]

master_backround

• Create new step MasterBackgroundNrsSlits step to handle NIRSpec MOS data in Spec2Pipeline [#5317]

• Implement option to save the 2d version of the calculated master background [#5317]

outlier_detection

• Fix bug where background was being subtracted on the input data [#4858]

pathloss

• Implement using a user-supplied correction which overrides all references. [#5302]

• Implement applying the inverse operation. [#5302]

photom

• Implement using a user-supplied correction which overrides all references. [#5302]

• Implement applying the inverse operation. [#5302]

pipeline

• Spec3Pipeline check whether master background subtraction has already occurred. [#5308]

• Implement master background subtraction in Spec2Pipeline for NIRSpec MOS data. [#5302]

• Include the per-slit failure traceback in any RuntimeError raised in Spec2Pipeline. [#5315]

ramp_fitting

• Reinstate copying of INT_TIMES table to output rateints product for TSO exposures. [#5321]

tso_photometry

• Fix a bug in the computation of integration time stamps when the INT_TIMES
  table is not available. [#5318]

JWST Build 7.6 rc1

Release Notes

align_refs

• Add bad pixel replacement for target and psf images [#4973]

assign_mtwcs

• Skip the step if any input MT_RA/DEC keyword values are missing. [#5015]

assign_wcs

• Enabled filteroffset correction for NIRISS and NIRCAM imaging modes. [#5018, #5027]

• Pass an optional input_frame parameter in assign_wcs.util.wcs_from_footprintss. [#5120]

• Improved calculation of bounding boxes in grism images. [#5122]

• Added two new optional parameters to utils.cerate_grism_bbox - wfss_extract_half_height
  and wavelength_range. [#5140]

• Shifted the bounding box of a resampled WCS by - 0.5 px to account for the
  center of the pixel. [#5241]

• Enable NIRSpec lamp processing in calspec2 pipeline. [#5267]

associations

• Update diagrams in documentation to change sloper to detector1. [#4986]

• Update level-3 rules to exclude IFU exposures from calwebb_tso3 associations. [#5202]

• Fix formatting error in Asn_IFUGrating product name construction. [#5231]

barshadow

• Correct bar shadow parity bug for yslit. [#5095]

combine_1d

• Skip spectra that are degenerate when combining [#5037]

cube_build

• Changed default weighting to 'emsm'. [#5277]

• Fixed formatting of NIRSpec s3d output product names. [#5231]

• Modified NIRSpec blotting to the find min and max ra and dec for each slice and only
  invert those values on slice that fall in range [#5144]

• Changed default weighting back to 'msm' until NIRSPEC cube pars ref file contains emsm info [#5134]

• Added checks read from cube pars reference file that parameters have valid data [#5134]

• Change the name of default cube type from world to skyalign [#4974]

• Add ifualign cubes to be cubes rotated on sky to align with ifu instrument plane [#4974]

• Change the name of MIRI alpha-beta cube type to internal_cal [#4974]

• Add ability to make NIRSpec internal_cal ifu cubes aligned with slicer plane [#4974]

• Change default weighting from msm to emsm [#4974]

• NIRSpec IFU cubes built from all wavelengths rather than those defined in cube par ref file [#4974]

• Removed wavelength planes that contained only 0 data. These planes are edge cases [#4974]

datamodels

• Add iscopy to ModelContainer init [#5256]

• Re-enable FITS-hash by default. [#5191]

• Add blend rule for keywords DETECTOR and MODULE. [#4998]

• Add methods Model.info and Model.search. [#4660]

• Trim MT_RA, MT_DEC keyword comments to fit within FITS record. [#4994]

• Add enum list and default value of 'NONE' for meta.instrument.lamp_mode [#5022]

• Add TIMEUNIT keyword to schemas. [#5109]

• Split pathloss object into pathloss_ps and pathloss_un in schemas. [#5112]

• Add "PERSISTENCE" DQ flag definition. [#5137]

• Fix nonsensical premature closing of FITS file of a DataModel. [#4930]

• Add a hash set/check to DataModel I/O to check whether schema traversal is necessary. [#5110]

• Update underlying MultiExposureModel from the SourceModelContainer models. [#5154]

• Add new MIRI LRS dither patterns to PATTTYPE enum list. [#5254]

extract_1d

• Implement aperture corrections in the Extract1dStep. [#4902]

• Fix bug in creating a polynomial fit used in background extraction. [#4970]

• Recheck the input model container in run_extract1d to select the correct processing [#5076]

• Rework/refactor many functions for style and readability. [#5079]

• Checks subwcs and new_slit variables exist before trying to delete them. [#5093]

• Parameter mmag_extract is now propagated to the extraction routine. [#5122]

• Updated the logic for when and how to use the source position to offset the
  location of the extraction regions specified in the EXTRACT1D reference file. [#5157]

• Fixed the conversion of flux to surface brightness for IFU extended source case [#5201]

• Fixed bugs in aperture correction for NIRSpec multi-slit modes. [#5260]

extract_2d

• Check that subwcs and new_slit variables exist before trying to delete them [#5093]

• Move NIRSpec wavecorr routines to the wavecorr step. [#5133]

• Added a new optional integer parameter to extract_2d (wfss_extract_half_height)
  which allows a user to specify the extraction height in the
  cross-dispersion direction for WFSS mode. [#5140]

flat_field

• For NIRSpec BOTS and ALLSLITS add the slit start corner to the subarray start corner
  when determining what region of the flat_field reference files to extract. [#5269]

• Enable NIRSpec lamp processing in calspec2 pipeline. [#5267]

fringe

• Update the fringe step to handle 3D inputs for MIRI MRS TSO mode. [#5202]

master_background

• Fix open files bug [#4995]

• Update to include pathloss corrections to NIRSpec IFU background [#5125]

mrs_imatch

• MRSIMatchStep to create its ModelContainers with iscopy=True [#5256]

outlier_detection

• Update median filter to use numpy's nanmedian. [#5114]

• Fix outlier_detection bug when saving intermediate results. [#5108]

• Update logic to correctly handle input CubeModel that have only
  1 integration. [#5211]

pathloss

• Fix bug in NIRSpec IFU data that causes valid pixel dq flags to set to
  NON-SCIENCE in the region of an overlapping bounding box slice [#5047]

• Update to save both point source and uniform source 2D pathloss correction
  arrays to output. [#5112]

persistence

• Flag pixels with high persistence using "PERSISTENCE" DQ flag instead
  of "DO_NOT_USE". [#5137]

pipeline

• Refactor the Image3Pipeline to use stpipe infrastructure. [#4990]

• Fix Coron3Pipeline to blend headers just from each input science model,
  not every integration. [#5007]

• Fix open files bug in get_config_from_reference class method, and in
  Spec2Pipeline, Spec3Pipeline and tso3. [#4995]

• Update calwebb_tso3 to do more robust checking of input data type.
  [#5107]

• Update the Spec2Pipeline to include the new wavecorr step and put
  srctype before wavecorr. [#5133]

• Update the Spec2Pipeline to skip extract_1d for IFU data that
  have not had a cube built (e.g. MIRI MRS TSO), and update the
  calwebb_tso-spec2.cfg configuration to turn on the fringe step
  and turn off cube_build for MIRI MRS TSO. [#5202]

• Update the Coron3Pipeline logic to correctly handle inputs that have
  only 1 integration. [#5211]

• Refactor Spec2Pipeline for execution logic and step flow isolation [#5214]

• Update Ami3Pipeline to only process psf and science members from the
  input ASN. [#5243]

• Enable NIRSpec lamp processing in calspec2 pipeline. [#5267]

photom

• Fix bug in NIRSpec IFU data that causes valid pixel dq flags to set to
  NON-SCIENCE in the region of an overlapping bounding box slice [#5047]

ramp_fitting

• Add multi-processing capability. [#4815]

• Fix crash when DRPFRMS1 is not set [#5096]

• Update to always create the rateints product, even when NINTS=1. [#5211]

resample_spec

• Fix artifacts in resampled NIRSpec slit data caused by NaNs in the WCS [#5217]

source_catalog

• Use gwcs.WCS instead of FITS WCS. [#5120]

• Changed the type of column is_star from float to bool. [#5140]

• Implemented algorithm for determining whether a source is a star.
  [#5234]

stpipe

• Limit reference file prefetch to the first "science" exptype
  when a pipeline has an association as input. [#5031]

• Remove further sloper references. [#4989]

• Enable prefetch of pars reference files for associations. [#5249]

transforms

• Wrap first spherical angle ("RA") at 360 degrees in the forward V23ToSky
  transformation and to 180 degrees for the inverse transformation ("V2").
  This is now done using models defined in astropy and gwcs packages
  replacing V23ToSky model in JWST's WCS pipeline. [#5206]

wavecorr

• Implemented the wavecorr step by pulling routines from the
  extract_2d step. [#5133]

JWST build 7.5

Release Notes

Highlights

The Level-1b script "set_telescope_pointing" has been updated to insert moving target coordinate keywords into level-1b product headers when the coordinate data is available in a moving target position table extension provided by SDP.

Association generator rules have been updated to:

• Limit NIRSpec IFU level 3 associations to contain data from a single grating+filter combination
• Produce level 2b and 3 associations for nodded observations for all spectroscopic modes
• Exclude dark exposures from all associations

Updates to Level-2a processing for all exposure modes include:

• Significant reduction in the memory footprint for processing of all exposure types

Updates to Level-2b processing for all exposure modes include:

• Populate ASNPOOL and ASNTABLE keywords for all products using an ASN to drive processing

Updates to Level-2b spectroscopic (calwebb_spec2) processing include:

• Bug fixes and refactoring of WCS transforms for WFSS exposures

Updates to the Level-3 imaging (calwebb_image3) processing include:

• Only load science and background members from an input ASN file
• The new capability to align image WCS to the Gaia catalog
• Added new capability to align moving target exposures in the moving target coordinate frame
• Source catalog generation has been completely rewritten to include new types of source measurements
• Updates to several steps to make them more robust to garbage data
• Many bug fixes to the creation of combined/resampled NIRSpec and MIRI LRS spectra

Updates to the Level-3 spectroscopic (calwebb_spec3) processing include:

• Update level-2b source-based products to include metadata from input exposures
• 1-D extracted products for WFSS mode now contain all data for a given source in a single x1d product
• Combined 1-D products for WFSS mode use the new product type suffix "c1d"
• Updated mrs_imatch to make it more robust to data that is out of specification

Updates to the Level-3 WFS&C (calwebb_wfs-image3) pipeline include:

• Updated the ASNTABLE keyword in all combined (wfscmb) image products

Full Release notes

assign_wcs

• Update keyword and attribute usage around SkyObject to reflect updated keywords. [#4943]
• Refactor PPS origin of NIRSpec MOS shutters from top left to bottom left. [#4959]
  associations
• Modify NIRSpec IFU level-3 ASN rules to include only one grating per association [#4926]

calwebb_coron3

• Update coron3 for new outlier detection application [#4968]

datamodels

• Add to_container to CubeModel to convert a cube to a list of images [#4968]
• Add getarray_noinit to DataModel to access arrays without causing initialization [#4968]
• Limit looping over HDU's while resolving arrays in schema [#4951]
• Relax asdf requirement and use validator flag when asdf 2.6.x is installed [#4905]
• Updated core schema to include recent Keyword Dictionary changes (remove TIME-END; add TDB-BEG, TDB-MID, TDB-END, XPOSURE, TELAPSE) [#4925]
• Populate meta.asn.table_name when an association is loaded into a ModelContainer. [#4873]

extract_1d

Add aperture correction in extract_1d processing. [#4902]

lib

Update SkyObject keys. [#4943]

mrs_imatch

Fix mrs_imatch to avoid calls to sigma_clipped_stats with all-zero arrays. [#4944]

photom

Fix flux units in photom for MultiSlit cases. [#4958]

pipeline

Updated calwebb_image3 pipeline to only load science and background member types from an input ASN. [#4937]
Updated the calwebb_spec2 pipeline to only use the basename of the source catalog file when updating the source_catalogue keyword for WFSS inputs. [#4940]

rscd

Fixed bug when the READPATT/SUBARRAY data is not found in RSCD reference file [#4934]

source_catalog

Add more concentration indices and update step docs. [#4906, #4908]
Added fallback background estimation method to make background estimation moare robust. [#4929]
Fixed the nearest-neighbor code to handle the case of exactly one detected source. [#4929]
Update abmag error calculation. [#4945]
Exit gracefully if APCORR ref file is missing. [#4948]

tweakreg

Added align_to_gaia processing as an option [#4599]