Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add StatisticsAggregator API #2554

Merged
merged 61 commits into from
Aug 2, 2024
Merged

Add StatisticsAggregator API #2554

merged 61 commits into from
Aug 2, 2024

Commits on Jul 22, 2024

  1. added stats extractor parent component

    added PlainExtractor based on numpy and scipy functions
    Tjark Miener authored and TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    e4585d8 View commit details
    Browse the repository at this point in the history
  2. added stats extractor based on sigma clipping

    Tjark Miener authored and TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    1e1afca View commit details
    Browse the repository at this point in the history
  3. added cut of outliers

    restructured the stats containers
    Tjark Miener authored and TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    d54aeda View commit details
    Browse the repository at this point in the history
  4. update docs

    Tjark Miener authored and TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    1ab0f1a View commit details
    Browse the repository at this point in the history
  5. formatted with black

    Tjark Miener authored and TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    074bd45 View commit details
    Browse the repository at this point in the history
  6. added pass for __call__ function

    Tjark Miener authored and TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    d1611ec View commit details
    Browse the repository at this point in the history
  7. Small commit for prototyping

    Christoph Toennis authored and TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    fdbb796 View commit details
    Browse the repository at this point in the history
  8. Removed unneeded functions

    Christoph Toennis authored and TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    76bb812 View commit details
    Browse the repository at this point in the history
  9. added unit tests

    Tjark Miener authored and TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    34902c4 View commit details
    Browse the repository at this point in the history
  10. added changelog

    Tjark Miener authored and TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    d25f85d View commit details
    Browse the repository at this point in the history
  11. fix lint

    Tjark Miener authored and TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    b5847f7 View commit details
    Browse the repository at this point in the history
  12. I altered the class variables to th evariance statistics extractor

    Christoph Toennis authored and TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    620ada6 View commit details
    Browse the repository at this point in the history
  13. added a container for mean variance images and fixed docustring

    Christoph Toennis authored and TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    4e275e7 View commit details
    Browse the repository at this point in the history
  14. I changed the container type for the StarVarianceExtractor

    Christoph Toennis authored and TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    9e68234 View commit details
    Browse the repository at this point in the history
  15. fix pylint

    Remove StarVarianceExtractor since is functionality is featured in the existing Extractors
    Tjark Miener authored and TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    cda269e View commit details
    Browse the repository at this point in the history
  16. change __call__() to _extract()

    Tjark Miener authored and TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    23c0536 View commit details
    Browse the repository at this point in the history
  17. minor renaming

    Tjark Miener authored and TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    5bd7cd9 View commit details
    Browse the repository at this point in the history
  18. use pytest.fixture for Extractors

    Tjark Miener authored and TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    ea3331b View commit details
    Browse the repository at this point in the history
  19. reduce duplicated code of the call function

    Tjark Miener authored and TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    9f61c56 View commit details
    Browse the repository at this point in the history
  20. Configuration menu
    Copy the full SHA
    36ffc88 View commit details
    Browse the repository at this point in the history
  21. added feature to shift the extraction sequence

    allow overlapping extraction sequences
    TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    a5caed4 View commit details
    Browse the repository at this point in the history
  22. fix boundary case for the last chunk

    renaming to chunk(s) and chunk_size and _shift
    
    added test for chunk_shift and boundary case
    TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    51d98f9 View commit details
    Browse the repository at this point in the history
  23. fix tests

    TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    5b15397 View commit details
    Browse the repository at this point in the history
  24. fix ruff

    TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    cb2a6fb View commit details
    Browse the repository at this point in the history
  25. edit docstring

    dl1 table is a Qtable from astropy and not a ndarray
    TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    a396d13 View commit details
    Browse the repository at this point in the history
  26. Configuration menu
    Copy the full SHA
    a08b831 View commit details
    Browse the repository at this point in the history
  27. fix typo

    TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    f20f1e3 View commit details
    Browse the repository at this point in the history
  28. edit docstring

    Length of table should be greater OR EQUAL than the size of chunk.
    TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    1be72cc View commit details
    Browse the repository at this point in the history
  29. removed lstchain relic

    The mask is needed in the lstchain flatfield class to calculate the relative gain, which we will do outside the StatisticsExtractor.
    TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    a072386 View commit details
    Browse the repository at this point in the history
  30. resolved remaining issues

    Check for outliers depends if one process flatfield (signal) or pedestal (noise) events.
    
    Change time to time_mono in order to read the telescope trigger and not the subarray.
    
    Fix tests to have the correct types.
    TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    9c8090e View commit details
    Browse the repository at this point in the history
  31. Configuration menu
    Copy the full SHA
    7ef82a5 View commit details
    Browse the repository at this point in the history
  32. Configuration menu
    Copy the full SHA
    09ed8dc View commit details
    Browse the repository at this point in the history
  33. Configuration menu
    Copy the full SHA
    3052f69 View commit details
    Browse the repository at this point in the history
  34. Configuration menu
    Copy the full SHA
    dbf6322 View commit details
    Browse the repository at this point in the history
  35. bug fix for chunks creation

    The for loop needs only run till length of the table minus the size of a chunk. Before the last chunks were created multiple times at the end.
    TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    643da4c View commit details
    Browse the repository at this point in the history
  36. bug fix treatment of last chunk and fix tests

    When the chunks are shifted (in the second pass) we want to drop overflow chunks, while for the first pass we want to keep the last overflow chunk and extract the stats for the last possible chunk.
    TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    6d7a9e5 View commit details
    Browse the repository at this point in the history
  37. edit comments

    TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    3087ff3 View commit details
    Browse the repository at this point in the history
  38. updated docs and tests

    TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    6147c0e View commit details
    Browse the repository at this point in the history
  39. properly name the fixture

    TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    537b092 View commit details
    Browse the repository at this point in the history
  40. update doc strings and variable names

    move outlier_check to traitlet; and remove redundant test on the input verification
    
    init extractor classes without fixture inside the test itself
    TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    362d336 View commit details
    Browse the repository at this point in the history
  41. use only one chunk iterator on the whole table

    before two chunk iterations were used for looping over two columns of the same table
    
    remove pylint command since not used in CI system of ctapipe
    TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    864a061 View commit details
    Browse the repository at this point in the history
  42. Configuration menu
    Copy the full SHA
    2d0d37d View commit details
    Browse the repository at this point in the history
  43. Configuration menu
    Copy the full SHA
    8f4f58c View commit details
    Browse the repository at this point in the history
  44. Configuration menu
    Copy the full SHA
    d09dc85 View commit details
    Browse the repository at this point in the history
  45. simplify _get_chunks

    polish doc string of __call__ function with explanation of overlapping and non-overlapping chunks as well as the overflow of the bounds of the dl1 table
    TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    71c051e View commit details
    Browse the repository at this point in the history
  46. generalized extract()

    moved extraction start and stop time outside stats container and only provide the extraction start, i.e. validity start. The returning of the caller is a dict with key the validity start and items the StatisticsContainer.
    TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    44ed0dc View commit details
    Browse the repository at this point in the history
  47. fix tests

    added check for exceptions
    TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    f2541f6 View commit details
    Browse the repository at this point in the history
  48. Configuration menu
    Copy the full SHA
    b772dff View commit details
    Browse the repository at this point in the history
  49. update docstring

    before we returned a list, but now a dict
    
    also update description of the returning dict
    TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    16914b6 View commit details
    Browse the repository at this point in the history
  50. return Table instead of dict

    added end of the chunk as well
    
    added also event IDs for the range
    TjarkMiener committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    2a62ab3 View commit details
    Browse the repository at this point in the history
  51. Configuration menu
    Copy the full SHA
    0ef25c6 View commit details
    Browse the repository at this point in the history

Commits on Jul 25, 2024

  1. fix random seed in tests

    TjarkMiener committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    051e66d View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    3b6b76b View commit details
    Browse the repository at this point in the history

Commits on Jul 31, 2024

  1. Configuration menu
    Copy the full SHA
    bc4156b View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    5a143dd View commit details
    Browse the repository at this point in the history

Commits on Aug 1, 2024

  1. Configuration menu
    Copy the full SHA
    cde6857 View commit details
    Browse the repository at this point in the history
  2. fix docs

    TjarkMiener committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    8fe3507 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    5275708 View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2024

  1. Configuration menu
    Copy the full SHA
    fbab2d1 View commit details
    Browse the repository at this point in the history
  2. fix typo

    remove second space
    TjarkMiener committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    98daeed View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    ea058a0 View commit details
    Browse the repository at this point in the history