Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add capability to run forecast in segments #2795

Merged

Commits on Aug 8, 2024

  1. Add capability to run forecast in segments

    Adds the ability to run a forecast in segments instead of all at
    once. To accomplish this, the `FHMIN_GFS` and `FHMAX_GFS` settings
    have been replaced as user-setable variables in favor of
    `FCST_SEGMENTS_STR_GFS`, a comma-separated list of the segment
    boundaries (thus there will be one more than the number of segments).
    For a traditional single-segment forecast, this would just be set to
    `"${FHMIN_GFS},${FHMAX_GFS}"`.
    
    The comma-separated list had to be used instead of a bash array as
    the variable must be exported in order for the rocoto generator to
    see it, and arrays cannot be exported from shell. Capabilty to parse
    these into python lists was added to wxflow in an accompanying PR.
    
    To accomodate the new segment metatasks that must be run serially,
    the capability of `create_task()` was expanded to allow a dictionary
    key of `is_serial`, which controls whether a metatask is parallel or
    serial using pre-existing capability in rocoto. The default when not
    given is parallel (i.e. most metatasks).
    
    Resolves NOAA-EMC#2274
    Refs NOAA-EMC/wxflow#39
    WalterKolczynski-NOAA committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    62435db View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    0254893 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    a7575b8 View commit details
    Browse the repository at this point in the history
  4. Change fcst segments to breakpoints, move to config.fcst

    Changes the way forecast segments are defined. Restores the original
    `FHMIN_GFS` and `FHMAX_GFS` and then adds a local `breakpnts` variable
    that contains the intermediate stopping points (if any). The original
    list of segment endpoints is then constructed from that.
    
    The determination of the `FHMIN` and `FHMAX` based on the segment is
    moved from `config.base` to `config.fcst`. This required adding some
    additional checks in `config.fcst` to clip other `FHMAX` variables to
    `FHMAX`.
    WalterKolczynski-NOAA committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    6ad1519 View commit details
    Browse the repository at this point in the history
  5. Update Test for GFS forecast segments

    Updates the extended GFS case to use forecast segments to test that
    capability for the GFS system (the GEFS case already tests segments
    as well).
    WalterKolczynski-NOAA committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    bf7a79c View commit details
    Browse the repository at this point in the history
  6. Adjust aerosol DA jobs for new config arrays

    Earlier update to parse comma-separated bash variables as lists in
    python mean we no longer need to do that in the task scripts.
    
    This commit is incomplete until a follow-up PR that will update the
    wxflow hash.
    WalterKolczynski-NOAA committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    9888bfa View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    1500a92 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    da62372 View commit details
    Browse the repository at this point in the history
  9. Update archive jinja to account for new lists

    With comma-separated lists now being read into python as lists, the
    jinja templates for the archive job had to be updated to not attempt
    to create them anymore.
    WalterKolczynski-NOAA committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    ca43f5a View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    db55ca2 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    6ae8fc6 View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    b2790b4 View commit details
    Browse the repository at this point in the history
  13. Correct prod dependency

    Missed updating one of the dependencies from task to metatask.
    WalterKolczynski-NOAA committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    e7ebe21 View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    7179c00 View commit details
    Browse the repository at this point in the history

Commits on Aug 10, 2024

  1. Configuration menu
    Copy the full SHA
    c73eecd View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    d0cd529 View commit details
    Browse the repository at this point in the history