Merge branch 'main' into CalibrationCalculators

.github/workflows/sonar.yml

@@ -58,7 +58,7 @@ jobs:
           cat coverage/sonar_env
 
       - name: Sonarqube Scan
-        uses: SonarSource/sonarqube-scan-action@v2
+        uses: SonarSource/sonarqube-scan-action@v3
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}

.mailmap

@@ -93,6 +93,8 @@ Anno Knierim <[email protected]> <[email protected]
 
 Vadym Voitsekhovskyi <[email protected]> <[email protected]>
 
+<[email protected]> <[email protected]>
+<[email protected]> <[email protected]>
 Tjark Miener <[email protected]>
 
 Michael Punch <[email protected]> <[email protected]>

AUTHORS

@@ -11,10 +11,12 @@ Lukas Beiske <[email protected]>
 Georg Schwefer <[email protected]>
 Jonas Hackfeld <[email protected]>
 Alison Mitchell <[email protected]>
+Tjark Miener <[email protected]>
 Dominik Neise <[email protected]>
 Michele Peresano <[email protected]>
 Christoph Deil <[email protected]>
 Justus Zorn <[email protected]>
+Christoph Toennis <[email protected]>
 Anno Knierim <[email protected]>
 Stefan Fröse <[email protected]>
 Rune Michael Dominik <[email protected]>
@@ -24,12 +26,12 @@ Franca Cassol <[email protected]>
 Clara Escanuela Nieves <[email protected]>
 Thomas Vuillaume <[email protected]>
 Kai Brügge <[email protected]>
+Mykhailo Dalchenko <[email protected]>
 Felix Werner <[email protected]>
 Satoshi Fukami <[email protected]>
 Michele Mastropietro <[email protected]>
 Jeremie DECOCK <[email protected]>
 Abelardo Moralejo Olaizola <[email protected]>
-Mykhailo Dalchenko <[email protected]>
 Tarek Hassan <[email protected]>
 Wrijupan Bhattacharyya <[email protected]>
 Francesco Visconti <[email protected]>
@@ -44,14 +46,14 @@ Tristan Carel <[email protected]>
 Daniel Nieto <[email protected]>
 Michael Punch <[email protected]>
 Miguel Nievas <[email protected]>
+Arnau Aguasca-Cabot <[email protected]>
 Christophe COSSOU <[email protected]>
 Cyril Alispach <[email protected]>
 Daniel Morcuende <[email protected]>
 Julien Lefaucheur <[email protected]>
 Konstantin Pfrang <[email protected]>
 Moritz Hütten <[email protected]>
 Thomas Armstrong <[email protected]>
-Tjark Miener <[email protected]>
 Alice Donini <[email protected]>
 Andrés Baquero <[email protected]>
 David Landriu <[email protected]>

CHANGES.rst

@@ -1,3 +1,60 @@
+ctapipe v0.22.0 (2024-09-12)
+============================
+
+API Changes
+-----------
+
+- The ``PointingInterpolator`` was moved from ``ctapipe.io`` to ``ctapipe.monitoring``. [`#2615 <https://github.com/cta-observatory/ctapipe/pull/2615>`__]
+
+
+Bug Fixes
+---------
+
+- Fix a redundant error message in ``Tool`` caused by normal ``SystemExit(0)`` [`#2575 <https://github.com/cta-observatory/ctapipe/pull/2575>`__]
+
+- Fix error message for non-existent config files. [`#2591 <https://github.com/cta-observatory/ctapipe/pull/2591>`__]
+
+
+New Features
+------------
+
+- ctapipe is now compatible with numpy 2.0. [`#2580 <https://github.com/cta-observatory/ctapipe/pull/2580>`__]
+  Note: not all new behaviour of numpy 2.0 is followed, as the core dependency ``numba`` does not yet implement
+  all changes from numpy 2.0. See `the numba announcement for more detail <https://numba.discourse.group/t/communicating-numpy-2-0-changes-to-numba-users/2457>`_.
+
+- Add lstchains image cleaning procedure including its pedestal cleaning method. [`#2541 <https://github.com/cta-observatory/ctapipe/pull/2541>`__]
+
+- A new ImageExtractor called ``VarianceExtractor`` was added
+  An Enum class was added to containers.py that is used in the metadata of the VarianceExtractor output [`#2543 <https://github.com/cta-observatory/ctapipe/pull/2543>`__]
+
+- Add API to extract the statistics from a sequence of images. [`#2554 <https://github.com/cta-observatory/ctapipe/pull/2554>`__]
+
+- The provenance system now records the reference metadata
+  of input and output files, if available. [`#2598 <https://github.com/cta-observatory/ctapipe/pull/2598>`__]
+
+- Add Interpolator class to generalize the PointingInterpolator in the monitoring collection. [`#2600 <https://github.com/cta-observatory/ctapipe/pull/2600>`__]
+
+- Add outlier detection components to identify faulty pixels. [`#2604 <https://github.com/cta-observatory/ctapipe/pull/2604>`__]
+
+- The ``ctapipe-merge`` tool now checks for duplicated input files and
+  raises an error in that case.
+
+  The ``HDF5Merger`` class, and thus also the ``ctapipe-merge`` tool,
+  now checks for duplicated obs_ids during merging, to prevent
+  invalid output files. [`#2611 <https://github.com/cta-observatory/ctapipe/pull/2611>`__]
+
+- The ``Instrument.site`` metadata item now accepts any string,
+  not just a pre-defined list of sites. [`#2616 <https://github.com/cta-observatory/ctapipe/pull/2616>`__]
+
+Refactoring and Optimization
+----------------------------
+
+- Update exception handling in tools
+
+  - Add a possibility to handle custom exception in ``Tool.run()``
+    with the preservation of the exit code. [`#2594 <https://github.com/cta-observatory/ctapipe/pull/2594>`__]
+
+
 ctapipe v0.21.2 (2024-06-26)
 ============================
 

docs/_static/switcher.json

@@ -9,6 +9,11 @@
     "version": "stable",
     "url": "https://ctapipe.readthedocs.io/en/stable/"
   },
+  {
+    "name": "v0.22.0",
+    "version": "v0.22.0",
+    "url": "https://ctapipe.readthedocs.io/en/v0.22.0/"
+  },
   {
     "name": "v0.21.2",
     "version": "v0.21.2",

docs/api-reference/monitoring/index.rst

@@ -18,10 +18,10 @@ Submodules
 
 .. toctree::
   :maxdepth: 1
-  :glob:
 
   aggregator
   calculator
+  interpolation
   outlier
 
 

docs/api-reference/monitoring/interpolation.rst

@@ -0,0 +1,11 @@
+.. _monitoring_interpolation:
+
+*************
+Interpolation
+*************
+
+
+Reference/API
+=============
+
+.. automodapi:: ctapipe.monitoring.interpolation

docs/changes/template.rst

@@ -1,6 +1,6 @@
 {% if render_title %}
 {% if versiondata.name %}
-{{ versiondata.name }} {{ versiondata.version }} ({{ versiondata.date }})
+{{ versiondata.name | lower }} {{ versiondata.version }} ({{ versiondata.date }})
 {{ top_underline * ((versiondata.name + versiondata.version + versiondata.date)|length + 4)}}
 {% else %}
 {{ versiondata.version }} ({{ versiondata.date }})

pyproject.toml

@@ -38,7 +38,7 @@ dependencies = [
     "joblib",
     "matplotlib ~=3.0",
     "numba >=0.56",
-    "numpy ~=1.16",
+    "numpy >=1.23,<3.0.0a0",
     "psutil",
     "pyyaml >=5.1",
     "requests",
@@ -48,6 +48,7 @@ dependencies = [
     "tqdm >=4.32",
     "traitlets ~=5.6",
     "zstandard",
+    "packaging",
 ]
 
 [project.optional-dependencies]

sonar-project.properties

@@ -5,4 +5,4 @@ sonar.python.version=3.10
 
 # ignore examples for coverage and issues, these are sphinx-gallery notebook scripts
 # which aren't really supported by sonarqube and lead to a lot of false positives
-sonar.exclusions=examples/**
+sonar.exclusions=examples/**,coverage/coverage.xml

src/ctapipe/atmosphere.py

@@ -19,6 +19,8 @@
 from astropy.table import QTable, Table
 from scipy.interpolate import interp1d
 
+from .compat import COPY_IF_NEEDED
+
 __all__ = [
     "AtmosphereDensityProfile",
     "ExponentialAtmosphereDensityProfile",
@@ -355,17 +357,19 @@ def __init__(self, table: Table):
     @u.quantity_input(height=u.m)
     def __call__(self, height) -> u.Quantity:
         log_density = self._density_interp(height.to_value(u.km))
-        return u.Quantity(10**log_density, DENSITY_UNIT, copy=False)
+        return u.Quantity(10**log_density, DENSITY_UNIT, copy=COPY_IF_NEEDED)
 
     @u.quantity_input(height=u.m)
     def integral(self, height) -> u.Quantity:
         log_col_density = self._col_density_interp(height.to_value(u.km))
-        return u.Quantity(10**log_col_density, GRAMMAGE_UNIT, copy=False)
+        return u.Quantity(10**log_col_density, GRAMMAGE_UNIT, copy=COPY_IF_NEEDED)
 
     @u.quantity_input(overburden=u.g / (u.cm * u.cm))
     def height_from_overburden(self, overburden) -> u.Quantity:
         log_overburden = np.log10(overburden.to_value(GRAMMAGE_UNIT))
-        return u.Quantity(self._height_interp(log_overburden), u.km, copy=False)
+        return u.Quantity(
+            self._height_interp(log_overburden), u.km, copy=COPY_IF_NEEDED
+        )
 
     def __repr__(self):
         return (

src/ctapipe/compat.py

@@ -3,6 +3,9 @@
 """
 import sys
 
+import numpy as np
+from packaging.version import Version
+
 __all__ = [
     "StrEnum",
 ]
@@ -15,3 +18,11 @@
 
     class StrEnum(str, Enum):
         """Compatibility backfill of StrEnum for python < 3.11"""
+
+
+# in numpy 1.x, copy=False allows copying if it cannot be avoided
+# in numpy 2.0, copy=False raises an error when the copy cannot be avoided
+# copy=None is a new option in numpy 2.0 for the previous behavior of copy=False
+COPY_IF_NEEDED = None
+if Version(np.__version__) < Version("2.0.0.dev"):
+    COPY_IF_NEEDED = False

src/ctapipe/conftest.py

@@ -592,6 +592,14 @@ def proton_train_clf(model_tmp_path, energy_regressor_path):
         ],
         raises=True,
     )
+
+    # modify obs_ids by adding a constant, this enables merging gamma and proton files
+    # which is used in the merge tool tests.
+    with tables.open_file(outpath, mode="r+") as f:
+        for table in f.walk_nodes("/", "Table"):
+            if "obs_id" in table.colnames:
+                obs_id = table.col("obs_id")
+                table.modify_column(colname="obs_id", column=obs_id + 1_000_000_000)
     return outpath
 
 

src/ctapipe/coordinates/camera_frame.py

@@ -15,6 +15,7 @@
     frame_transform_graph,
 )
 
+from ..compat import COPY_IF_NEEDED
 from .representation import PlanarRepresentation
 from .telescope_frame import TelescopeFrame
 
@@ -144,10 +145,14 @@ def camera_to_telescope(camera_coord, telescope_frame):
     # where theta is the angle to the optical axis and r is the distance
     # to the camera center in the focal plane
     fov_lat = u.Quantity(
-        (x_rotated / focal_length).to_value(u.dimensionless_unscaled), u.rad, copy=False
+        (x_rotated / focal_length).to_value(u.dimensionless_unscaled),
+        u.rad,
+        copy=COPY_IF_NEEDED,
     )
     fov_lon = u.Quantity(
-        (y_rotated / focal_length).to_value(u.dimensionless_unscaled), u.rad, copy=False
+        (y_rotated / focal_length).to_value(u.dimensionless_unscaled),
+        u.rad,
+        copy=COPY_IF_NEEDED,
     )
 
     representation = UnitSphericalRepresentation(lat=fov_lat, lon=fov_lon)

src/ctapipe/core/tool.py

@@ -150,15 +150,15 @@ def main():
         trait=Path(
             exists=True,
             directory_ok=False,
-            help=(
-                "List of configuration files with parameters to load "
-                "in addition to command-line parameters. "
-                "The order listed is the order of precedence (later config parameters "
-                "overwrite earlier ones), however parameters specified on the "
-                "command line always have the highest precedence. "
-                "Config files may be in JSON, YAML, TOML, or Python format"
-            ),
-        )
+        ),
+        help=(
+            "List of configuration files with parameters to load "
+            "in addition to command-line parameters. "
+            "The order listed is the order of precedence (later config parameters "
+            "overwrite earlier ones), however parameters specified on the "
+            "command line always have the highest precedence. "
+            "Config files may be in JSON, YAML, TOML, or Python format"
+        ),
     ).tag(config=True)
 
     log_config = Dict(default_value={}).tag(config=True)