Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for forecast-only runs on AWS #2711

Merged
merged 71 commits into from
Aug 13, 2024
Merged

Add support for forecast-only runs on AWS #2711

merged 71 commits into from
Aug 13, 2024

Commits on Jun 18, 2024

  1. compiled OK now

    @weihuang-jedi
    weihuang-jedi committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    103f2c4 View commit details
    Browse the repository at this point in the history

Commits on Jun 19, 2024

  1. Merge branch 'NOAA-EMC:develop' into aws-forecast-only

    @weihuang-jedi
    weihuang-jedi authored Jun 19, 2024
    Configuration menu
    Copy the full SHA
    916ff6c View commit details
    Browse the repository at this point in the history

  2. re-test on aws with fewer changes

    @weihuang-jedi
    weihuang-jedi committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    b0ac406 View commit details
    Browse the repository at this point in the history

Commits on Jun 21, 2024

  1. make change in tasks.py to avoid error finding libiomp5.so problem

    @weihuang-jedi
    weihuang-jedi committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    3de972f View commit details
    Browse the repository at this point in the history

  2. Merge branch 'NOAA-EMC:develop' into aws-forecast-only

    @weihuang-jedi
    weihuang-jedi authored Jun 21, 2024
    Configuration menu
    Copy the full SHA
    8308375 View commit details
    Browse the repository at this point in the history

Commits on Jun 22, 2024

  1. add comments so the reviewers know that these changes are for AWS, an… 

    …d hope can provide a better way
    @weihuang-jedi
    weihuang-jedi committed Jun 22, 2024
    Configuration menu
    Copy the full SHA
    bc4c4a8 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'aws-forecast-only' of ssh://github.com/NOAA-EPIC/global… 

    …-workflow-cloud into aws-forecast-only
    @weihuang-jedi
    weihuang-jedi committed Jun 22, 2024
    Configuration menu
    Copy the full SHA
    924aede View commit details
    Browse the repository at this point in the history

  3. add comments so the reviewers know that these changes are for AWS, an… 

    …d hope can provide a better way
    @weihuang-jedi
    weihuang-jedi committed Jun 22, 2024
    Configuration menu
    Copy the full SHA
    b724937 View commit details
    Browse the repository at this point in the history

Commits on Jun 25, 2024

  1. reverse config.resource changes, and memory restriction on AWS

    @weihuang-jedi
    weihuang-jedi committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    12ab29f View commit details
    Browse the repository at this point in the history

  2. sync with emc repo

    @weihuang-jedi
    weihuang-jedi committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    adff250 View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2024

  1. move common data to a shared place

    @weihuang-jedi
    weihuang-jedi committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    2290ea2 View commit details
    Browse the repository at this point in the history

  2. use ICs from s3-bucket

    @weihuang-jedi
    weihuang-jedi committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    cd2c8e7 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'develop' into aws-forecast-only

    @weihuang-jedi
    weihuang-jedi authored Jun 26, 2024
    Configuration menu
    Copy the full SHA
    4e144e5 View commit details
    Browse the repository at this point in the history

Commits on Jul 2, 2024

  1. change as suggested by reviewer

    @weihuang-jedi
    weihuang-jedi committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    46e3ef5 View commit details
    Browse the repository at this point in the history

  2. sync with develop

    @weihuang-jedi
    weihuang-jedi committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    32f13eb View commit details
    Browse the repository at this point in the history

Commits on Jul 4, 2024

  1. sync sorc/ufs_model.fd

    @weihuang-jedi
    weihuang-jedi committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    a34a4c8 View commit details
    Browse the repository at this point in the history

  2. remove mpmd_opt from APRUN_UFS

    @weihuang-jedi
    weihuang-jedi committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    44011a3 View commit details
    Browse the repository at this point in the history

Commits on Jul 5, 2024

  1. mpmd_opt and switch off tracker/genesis default for AWS

    @weihuang-jedi
    weihuang-jedi committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    965ec80 View commit details
    Browse the repository at this point in the history

  2. add TODO

    @weihuang-jedi
    weihuang-jedi committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    3ce268e View commit details
    Browse the repository at this point in the history

Commits on Jul 6, 2024

  1. remove ncl version on AWS

    @weihuang-jedi
    weihuang-jedi committed Jul 6, 2024
    Configuration menu
    Copy the full SHA
    f03ac78 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/develop' into aws-forecast-only

    @weihuang-jedi
    weihuang-jedi committed Jul 6, 2024
    Configuration menu
    Copy the full SHA
    007a56b View commit details
    Browse the repository at this point in the history

  3. sync ufs_model

    @weihuang-jedi
    weihuang-jedi committed Jul 6, 2024
    Configuration menu
    Copy the full SHA
    2f6ec6e View commit details
    Browse the repository at this point in the history

Commits on Jul 10, 2024

  1. Merge branch 'NOAA-EMC:develop' into aws-forecast-only

    @weihuang-jedi
    weihuang-jedi authored Jul 10, 2024
    Configuration menu
    Copy the full SHA
    dba83a7 View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2024

  1. Merge branch 'NOAA-EMC:develop' into aws-forecast-only

    @weihuang-jedi
    weihuang-jedi authored Jul 12, 2024
    Configuration menu
    Copy the full SHA
    24fe804 View commit details
    Browse the repository at this point in the history

  2. sync and remove gempak from noaacloud

    @weihuang-jedi
    weihuang-jedi committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    e8a2e0f View commit details
    Browse the repository at this point in the history

Commits on Jul 15, 2024

  1. Merge branch 'NOAA-EMC:develop' into aws-forecast-only

    @weihuang-jedi
    weihuang-jedi authored Jul 15, 2024
    Configuration menu
    Copy the full SHA
    4013eb1 View commit details
    Browse the repository at this point in the history

  2. update modules hash

    @weihuang-jedi
    weihuang-jedi committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    a548c7f View commit details
    Browse the repository at this point in the history

  3. update module hash

    @weihuang-jedi
    weihuang-jedi committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    d37e646 View commit details
    Browse the repository at this point in the history

Commits on Jul 17, 2024

  1. use bucket

    @weihuang-jedi
    weihuang-jedi committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    2a80162 View commit details
    Browse the repository at this point in the history

  2. remove /scratch1, but kept TODO

    @weihuang-jedi
    weihuang-jedi committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    fa44862 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'NOAA-EMC:develop' into aws-forecast-only

    @weihuang-jedi
    weihuang-jedi authored Jul 17, 2024
    Configuration menu
    Copy the full SHA
    55c7e7e View commit details
    Browse the repository at this point in the history

Commits on Jul 19, 2024

  1. re-sync

    @weihuang-jedi
    weihuang-jedi committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    07851dc View commit details
    Browse the repository at this point in the history

  2. sync

    @weihuang-jedi
    weihuang-jedi committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    492808d View commit details
    Browse the repository at this point in the history

Commits on Jul 23, 2024

  1. add is_exclusive to resource.AWSPW

    @weihuang-jedi
    weihuang-jedi committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    d7a262e View commit details
    Browse the repository at this point in the history

  2. sync hash with EMC repo

    @weihuang-jedi
    weihuang-jedi committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    af573af View commit details
    Browse the repository at this point in the history

  3. remove --export=ALL from native, when is_exclusive set true

    @weihuang-jedi
    weihuang-jedi committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    0929180 View commit details
    Browse the repository at this point in the history

  4. sync

    @weihuang-jedi
    weihuang-jedi committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    06fecca View commit details
    Browse the repository at this point in the history

Commits on Jul 25, 2024

  1. Merge branch 'NOAA-EMC:develop' into aws-forecast-only

    @weihuang-jedi
    weihuang-jedi authored Jul 25, 2024
    Configuration menu
    Copy the full SHA
    d8783ab View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/develop' into aws-forecast-only

    @weihuang-jedi
    weihuang-jedi committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    d22bc6d View commit details
    Browse the repository at this point in the history

  3. Merge branch 'aws-forecast-only' of ssh://github.com/NOAA-EPIC/global… 

    …-workflow-cloud into aws-forecast-only
    @weihuang-jedi
    weihuang-jedi committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    a5c441f View commit details
    Browse the repository at this point in the history

  4. Make AWS works similar to on-prem machine

    @weihuang-jedi
    weihuang-jedi committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    77e8233 View commit details
    Browse the repository at this point in the history

  5. remove --export=ALL from 'native'

    @weihuang-jedi
    weihuang-jedi committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    96f73ba View commit details
    Browse the repository at this point in the history

  6. remove --export=ALL from 'native'

    @weihuang-jedi
    weihuang-jedi committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    a33a3be View commit details
    Browse the repository at this point in the history

  7. Merge branch 'NOAA-EMC:develop' into aws-forecast-only

    @weihuang-jedi
    weihuang-jedi authored Jul 25, 2024
    Configuration menu
    Copy the full SHA
    80b294b View commit details
    Browse the repository at this point in the history

  8. add py-f90nml to noaacloud modulefile

    @weihuang-jedi
    weihuang-jedi committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    01a8928 View commit details
    Browse the repository at this point in the history

  9. remove un-necessary added lines

    @weihuang-jedi
    weihuang-jedi committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    b035947 View commit details
    Browse the repository at this point in the history

  10. remove un-necessary added lines

    @weihuang-jedi
    weihuang-jedi committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    bf3b460 View commit details
    Browse the repository at this point in the history

Commits on Jul 26, 2024

  1. remove added lines which was originally for AWS, but should be define… 

    …d in hosts/awspw.yaml
    @weihuang-jedi
    weihuang-jedi committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    47627ff View commit details
    Browse the repository at this point in the history

  2. restore as develop

    @weihuang-jedi
    weihuang-jedi committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    7bf8900 View commit details
    Browse the repository at this point in the history

Commits on Jul 29, 2024

  1. try to fix pynorms error

    @weihuang-jedi
    weihuang-jedi committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    0685a8f View commit details
    Browse the repository at this point in the history

  2. Merge branch 'NOAA-EMC:develop' into aws-forecast-only

    @weihuang-jedi
    weihuang-jedi authored Jul 29, 2024
    Configuration menu
    Copy the full SHA
    381403d View commit details
    Browse the repository at this point in the history

  3. Merge branch 'aws-forecast-only' of ssh://github.com/NOAA-EPIC/global… 

    …-workflow-cloud into aws-forecast-only
    @weihuang-jedi
    weihuang-jedi committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    0e71f7d View commit details
    Browse the repository at this point in the history

Commits on Jul 30, 2024

  1. Merge branch 'NOAA-EMC:develop' into aws-forecast-only

    @weihuang-jedi
    weihuang-jedi authored Jul 30, 2024
    Configuration menu
    Copy the full SHA
    2024835 View commit details
    Browse the repository at this point in the history

  2. sync with EMC repo

    @weihuang-jedi
    weihuang-jedi committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    2c52016 View commit details
    Browse the repository at this point in the history

  3. sync Gaea link with EMC repo, and only include blocks/packs that run … 

    …on AWS
    @weihuang-jedi
    weihuang-jedi committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    cd6c541 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'aws-forecast-only' of github.com:NOAA-EPIC/global-workf… 

    …low-cloud into aws-forecast-only
    @weihuang-jedi
    weihuang-jedi committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    1f60ed0 View commit details
    Browse the repository at this point in the history

  5. merge fro develop

    @weihuang-jedi
    weihuang-jedi committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    e1a57b4 View commit details
    Browse the repository at this point in the history

Commits on Jul 31, 2024

  1. Remove ACCOUNT_SERVICE

    @weihuang-jedi
    weihuang-jedi committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    fe9a457 View commit details
    Browse the repository at this point in the history

Commits on Aug 1, 2024

  1. Merge branch 'NOAA-EMC:develop' into aws-forecast-only

    @weihuang-jedi
    weihuang-jedi authored Aug 1, 2024
    Configuration menu
    Copy the full SHA
    5c6e052 View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2024

  1. Merge branch 'NOAA-EMC:develop' into aws-forecast-only

    @weihuang-jedi
    weihuang-jedi authored Aug 2, 2024
    Configuration menu
    Copy the full SHA
    93b1e66 View commit details
    Browse the repository at this point in the history

  2. correct pynorms error

    @weihuang-jedi
    weihuang-jedi committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    f900893 View commit details
    Browse the repository at this point in the history

Commits on Aug 5, 2024

  1. Merge branch 'NOAA-EMC:develop' into aws-forecast-only

    @weihuang-jedi
    weihuang-jedi authored Aug 5, 2024
    Configuration menu
    Copy the full SHA
    f599cd7 View commit details
    Browse the repository at this point in the history

Commits on Aug 6, 2024

  1. Update workflow/rocoto/workflow_xml.py 

    Co-authored-by: David Huber <[email protected]>
    @weihuang-jedi @DavidHuber-NOAA
    weihuang-jedi and DavidHuber-NOAA authored Aug 6, 2024
    Configuration menu
    Copy the full SHA
    1ee5492 View commit details
    Browse the repository at this point in the history

  2. fix pynorms issues

    @weihuang-jedi
    weihuang-jedi committed Aug 6, 2024
    Configuration menu
    Copy the full SHA
    6d6231a View commit details
    Browse the repository at this point in the history

  3. fix pynorms issues

    @weihuang-jedi
    weihuang-jedi committed Aug 6, 2024
    Configuration menu
    Copy the full SHA
    eb262be View commit details
    Browse the repository at this point in the history

  4. only one pycodestyle error left now

    @weihuang-jedi
    weihuang-jedi committed Aug 6, 2024
    Configuration menu
    Copy the full SHA
    0db930d View commit details
    Browse the repository at this point in the history

  5. pycodestype passed without any error

    @weihuang-jedi
    weihuang-jedi committed Aug 6, 2024
    Configuration menu
    Copy the full SHA
    06093af View commit details
    Browse the repository at this point in the history

  6. Merge branch 'NOAA-EMC:develop' into aws-forecast-only

    @weihuang-jedi
    weihuang-jedi authored Aug 6, 2024
    Configuration menu
    Copy the full SHA
    6fff724 View commit details
    Browse the repository at this point in the history

Commits on Aug 7, 2024

  1. Merge branch 'NOAA-EMC:develop' into aws-forecast-only

    @weihuang-jedi
    weihuang-jedi authored Aug 7, 2024
    Configuration menu
    Copy the full SHA
    f23d2d0 View commit details
    Browse the repository at this point in the history

Commits on Aug 8, 2024

  1. Merge branch 'NOAA-EMC:develop' into aws-forecast-only

    @weihuang-jedi
    weihuang-jedi authored Aug 8, 2024
    Configuration menu
    Copy the full SHA
    bd1c954 View commit details
    Browse the repository at this point in the history

Commits on Aug 12, 2024

  1. Merge branch 'NOAA-EMC:develop' into aws-forecast-only

    @weihuang-jedi
    weihuang-jedi authored Aug 12, 2024
    Configuration menu
    Copy the full SHA
    d12d9e9 View commit details
    Browse the repository at this point in the history