Problems processing data from 20240626

[marialainez]

We are getting an error in the r0_to_dl1 step for various subruns of 20240626 (e.g. 17851.0075) with which we do not know how to proceed. This is the error traceback that we are getting:

WARNING: non-ASCII characters are present in the FITS file header and have been replaced by "?" characters [astropy.io.fits.util]
non-ASCII characters are present in the FITS file header and have been replaced by "?" characters
WARNING: VerifyWarning: Error validating header for HDU #4 (note: Astropy uses zero-based indexing).
    Header size is not multiple of 2880: 880335602
There may be extra bytes after the last HDU or the file is corrupted. [astropy.io.fits.hdu.hdulist]
VerifyWarning: Error validating header for HDU #4 (note: Astropy uses zero-based indexing).
    Header size is not multiple of 2880: 880335602
There may be extra bytes after the last HDU or the file is corrupted.
Traceback (most recent call last):
  File "/fefs/aswg/software/conda/envs/osa-v0.10/bin/lstchain_data_r0_to_dl1", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/fefs/aswg/software/conda/envs/osa-v0.10/lib/python3.11/site-packages/lstchain/scripts/lstchain_data_r0_to_dl1.py", line 230, in main
    r0_to_dl1.r0_to_dl1(
  File "/fefs/aswg/software/conda/envs/osa-v0.10/lib/python3.11/site-packages/lstchain/reco/r0_to_dl1.py", line 352, in r0_to_dl1
    source = EventSource(input_url=input_filename,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fefs/aswg/software/conda/envs/osa-v0.10/lib/python3.11/site-packages/ctapipe_io_lst/__init__.py", line 304, in __init__
    self.multi_file = MultiFiles(
                      ^^^^^^^^^^^
  File "/fefs/aswg/software/conda/envs/osa-v0.10/lib/python3.11/site-packages/ctapipe_io_lst/multifiles.py", line 123, in __init__
    self._load_next_subrun(stream)
  File "/fefs/aswg/software/conda/envs/osa-v0.10/lib/python3.11/site-packages/ctapipe_io_lst/multifiles.py", line 184, in _load_next_subrun
    event = next(self._events_tables[stream])
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fefs/aswg/software/conda/envs/osa-v0.10/lib/python3.11/site-packages/protozfits/__init__.py", line 153, in __next__
    raise StopIteration
StopIteration

And here is an example of the command that is launched:

lstchain_data_r0_to_dl1 --input-file=/fefs/aswg/data/real/R0G/20240626/LST-1.1.Run17851.0075.fits.fz --output-dir=/fefs/aswg/data/real/running_analysis/20240626/v0.10 --pedestal-file=/fefs/aswg/data/real/monitoring/PixelCalibration/Cat-A/drs4_baseline/20240626/v0.10.10/drs4_pedestal.Run17846.0000.h5 --calibration-file=/fefs/aswg/data/real/monitoring/PixelCalibration/Cat-A/calibration/20240626/v0.10.10/calibration_filters_52.Run17847.0000.h5 --time-calibration-file=/fefs/aswg/data/real/monitoring/PixelCalibration/Cat-A/drs4_time_sampling_from_FF/20231102/v0.10.4/time_calibration.Run15253.0000.h5 --systematic-correction-file=/fefs/aswg/data/real/monitoring/PixelCalibration/Cat-A/ffactor_systematics/20230410/v0.10.3/calibration_scan_fit_20230410.0000.h5 --config=/fefs/aswg/lstosa/config_files_lstchain/lstchain_standard_v0.10_no_heuristic_ff.json --pointing-file=/fefs/onsite/monitoring/driveLST1/DrivePositioning/DrivePosition_log_20240626.txt --run-summary-path=/fefs/aswg/data/real/monitoring/RunSummary/RunSummary_20240626.ecsv

NOTE: input files are already gain selected

Software settings:

(osa-v0.10) [lstanalyzer@tcs06 ~]$ conda list lst
# packages in environment at /fefs/aswg/software/conda/envs/osa-v0.10:
#
# Name                    Version                   Build  Channel
ctapipe-io-lst            0.23.2                   pypi_0    pypi
lstchain                  0.10.10                  pypi_0    pypi

@moralejo @maxnoe

[maxnoe]

This is the same error we discussed some time ago via email: there are no events in the input file and it wasn't properly by the zfits writer.

[morcuended]

Apparently, this problem does not happen when directly using the R0 data, but only after having done first the gain selection.

[maxnoe]

MMmh. Then I think the only reason can be that the original data only contained invalid (event_id == 0, empty events) that are not written out by the gain selector tool.

I think the gain selector tool fails in such a case, or it at least should fail in such a case and OSA should probably check that when running it.

[morcuended]

Gain selection tool did not fail