Implement global-workflow on AWS #2549

weihuang-jedi · 2024-04-27T01:49:18Z

PRs should meet these guidelines:
- Each PR should address ONE topic and have an associated issue.
- No hard-coded paths or personal directories.
- No temporary or backup files should be committed (including logs).
- Any code that you disabled by being commented out should be removed or reenabled.

Please delete all these comments before submitting the PR.

-->

Description

This PR implement global-workflow on AWS.

Resolves NOAA-EPIC#2

Type of change

New feature (adds functionality)

Change characteristics

Does this change require a documentation update? YES

How has this been tested?

Tested on AWS

Example:

Clone and build on AWS

Checklist

[x ] Code changes for GSI-Utils
[ x] Code changes for UFS-Utils
[ x] Code Changes for GFS-Utils
[x ] Code changes for GSI-monitor
[x ] Code changes for UPP
[x ] Code changes for UFS-weather-model
x[ ] Code changes for GSI
[x ] Code changes for GDAS

…cloud into wei-epic-aws

…loud into wei-epic-aws

weihuang-jedi · 2024-04-27T02:01:26Z

This is at its early stage, depends on many things. Will mark it ready when things all settled.

…loud into wei-epic-aws

…flow-cloud into wei-epic-aws

…loud into wei-epic-aws

…flow-cloud into wei-epic-aws

…loud into wei-epic-aws

aerorahul

Please describe the hard to understand updates in this PR.
Please see specific comments.

aerorahul · 2024-06-14T22:05:46Z

.gitmodules

@@ -1,6 +1,6 @@
 [submodule "sorc/ufs_model.fd"]
  path = sorc/ufs_model.fd
-  url = https://github.com/ufs-community/ufs-weather-model
+  url = https://github.com/ufs-community/ufs-weather-model.git


please revert.

aerorahul · 2024-06-14T22:06:22Z

env/AZUREPW.env

@@ -0,0 +1,298 @@
+#! /usr/bin/env bash


Please remove Azure and Google additions from this PR as this PR is titled "Implemente global-workflow on AWS"

aerorahul · 2024-06-14T22:08:25Z

parm/config/gfs/config.base

+#export DO_TRACKER="@DO_TRACKER@"         # Hurricane track verification
+#export DO_GENESIS="@DO_GENESIS@"         # Cyclone genesis verification
+#export DO_GENESIS_FSU="@DO_GENESIS_FSU@" # Cyclone genesis verification (FSU)
+export DO_TRACKER="NO"         # Hurricane track verification
+export DO_GENESIS="NO"         # Cyclone genesis verification
+export DO_GENESIS_FSU="NO" # Cyclone genesis verification (FSU)


Please revert.

aerorahul · 2024-06-14T22:09:07Z

parm/config/gfs/config.resources

+  "AZUREPW")
+    export PARTITION_BATCH="compute"
+    npe_node_max=24
+    ;;
+  "GOOGLEPW")
+    export PARTITION_BATCH="compute"
+    npe_node_max=30


Please withhold till these platforms are available and the workflow is being ported to them.

aerorahul · 2024-06-14T22:09:22Z

parm/config/gfs/config.resources

@@ -804,7 +812,7 @@ case ${step} in
    ;;

  "atmos_products")
-    export wtime_atmos_products="00:15:00"
+    export wtime_atmos_products="01:15:00"


This is a significant increase.
Please revert.

aerorahul · 2024-06-14T22:16:39Z

workflow/hosts/googlepw.yaml

@@ -0,0 +1,24 @@
+BASE_GIT: '/work/noaa/global/glopara/git'


Please remove this file.
These are orion paths

aerorahul · 2024-06-14T22:16:57Z

workflow/rocoto/gfs_tasks.py

+            pslot = self._base['PSLOT']
+            if ( 'BASE_CPLIC' in cpl_ic.keys() ):
+                base_cplic = f"{cpl_ic['BASE_CPLIC']}"
+            else:
+                base_cplic = os.environ.get('BASE_CPLIC')
+            if ( 'CPL_ATMIC' in cpl_ic.keys() ):
+                cpl_atmic = f"{cpl_ic['CPL_ATMIC']}"
+            else:
+                cpl_atmic = os.environ.get('CPL_ATMIC')
+
+            prefix = f"{base_cplic}/{cpl_atmic}/@Y@m@d@H/atmos"
+
+            pw_csp = os.environ.get('PW_CSP')
+            use_ufs_utils_format = os.environ.get('USE_UFS_UTILS_FORMAT', False)
+            if ( pw_csp in ['aws', 'azure', 'google'] or use_ufs_utils_format):
+                icdir = f"{base_cplic}/{cpl_atmic}"
+
+                if('IC_PREFIX' in cpl_ic.keys()):
+                    cpl_ic_prefix = f"{icdir}/{cpl_ic['IC_PREFIX']}"
+                else:
+                    cpl_ic_prefix = 'gfs'
+                if('IC_TYPE'  in cpl_ic.keys()):
+                    cpl_ic_type = f"{cpl_ic['IC_TYPE']}"
+                else:
+                    cpl_ic_type = 'input'
+                prefix = f"{icdir}/{cpl_ic_prefix}.@Y@m@d/@H/model_data/atmos/{cpl_ic_type}"


This should not be needed.

aerorahul · 2024-06-14T22:17:50Z

workflow/rocoto/rocoto.py

+    pw_csp = os.environ.get('PW_CSP')
+    if pw_csp not in ['aws', 'azure', 'google']:


rocoto.py need not know if this is PW_CSP.
Please outline the issue and we can discuss.

aerorahul · 2024-06-14T22:18:04Z

workflow/rocoto/tasks.py

+            pw_csp = os.environ.get('PW_CSP')
+            if ( pw_csp in ['aws', 'azure', 'google'] ):
+                native = '--export=ALL --exclusive'
+            else:
+                native = '--export=NONE'


This should be handled elsewhere.

aerorahul · 2024-06-14T22:18:34Z

workflow/rocoto/workflow_xml.py

+                  f'#################### {pslot} ####################',
+                  f'MAILTO="{replyto}"'
+                  ]
+        pw_csp = os.environ.get('PW_CSP')
+        if ( pw_csp in ['aws', 'azure', 'google'] ):
+             strings = np.append(strings,
+                       [
+                       f'SHELL="/bin/bash"',
+                       f'BASH_ENV="/etc/bashrc"'
+                       ])
+        strings = np.append(strings,
+                  [
+                  f'{cronintstr} {rocotorunstr}',
+                  '#################################################################',
+                  ''])


Can you describe the reason here

HenryRWinterbottom · 2024-06-14T23:18:43Z

@weihuang-jedi Please properly follow the PR template guidelines. See, for example, #2688.

DavidHuber-NOAA · 2024-06-25T14:51:19Z

Should this be closed in favor of #2711?

aerorahul · 2024-06-25T16:57:45Z

Closing in favor of #2711

weihuang-jedi and others added 21 commits April 8, 2024 16:48

add noaacloud module for AWS

b3b3b17

Merge branch 'develop' of ssh://github.com/NOAA-EPIC/global-workflow-…

1c5231a

…cloud into wei-epic-aws

to compile on AWS

149d776

check back on hera to make sure it did not break anything

2fb6e6d

add mods to save aws changes

30b615a

mv .aws.tar to mods4aws

5ec3a78

update gdas modulefile

43a6b33

Merge remote-tracking branch 'origin' into wei-epic-aws

4512075

add changes of gsi_enkf

d84fded

add two vers for noaacloud

534cbb3

prepare to run case

e9d41a0

re-add these tar files

b5eb58c

update gsi_utils module file

7e2d1c4

update version and modules for AWS

c39eb7e

add hosts info for noaacloud

47bffad

compiled OK for ca-sfc-emc account

956a3e5

add host and env for CSP

2768686

Merge remote-tracking branch 'origin' into wei-epic-aws

6404639

compile gdas

cab4490

Merge branch 'wei-epic-aws' of github.com:NOAA-EPIC/global-workflow-c…

e6903e1

…loud into wei-epic-aws

Merge branch 'NOAA-EMC:develop' into wei-epic-aws

41eb077

weihuang-jedi and others added 8 commits April 27, 2024 02:14

Merge branch 'wei-epic-aws' of github.com:NOAA-EPIC/global-workflow-c…

4e457a1

…loud into wei-epic-aws

run through at ca-epic

5e588f0

Merge branch 'wei-epic-aws' of github.com:NOAA-EPIC/global-workflow-c…

db17544

…loud into wei-epic-aws

sync to ca-sfc-emc

2cc1c56

Merge remote-tracking branch 'origin' into wei-epic-aws

bee6fa9

testing on ca-sfc-emc account

c310b5e

Merge branch 'NOAA-EMC:develop' into wei-epic-aws

d388ad4

clean up some temporary files

c18cc33

weihuang-jedi added 22 commits May 28, 2024 21:31

merge develop into wei-epic-aws

529e258

merge develop into wei-epic-aws

c690fee

change google CPU numbers

75af7b7

Merge branch 'wei-epic-aws' of github.com:NOAA-EPIC/global-workflow-c…

22ab9a5

…loud into wei-epic-aws

tidying up AWS changes

f9c4fc7

handle ICs differently on AWS

f2667d8

handle ICs differently on AWS

b8d3ef1

tidying up

d0b8b50

continuing tidy up

f6f3722

Merge branch 'wei-epic-aws' of ssh://github.com/NOAA-EPIC/global-work…

99ee72d

…flow-cloud into wei-epic-aws

switch ufs-weather-model back

613b0fc

add noaacloud to compile gdas

31e3b0c

before final sync with EMC repo

9c8e01a

Merge branch 'wei-epic-aws' of github.com:NOAA-EPIC/global-workflow-c…

c120f37

…loud into wei-epic-aws

sync with develop

1079952

make it also run on hera

42f9990

Merge branch 'wei-epic-aws' of ssh://github.com/NOAA-EPIC/global-work…

eec9c5a

…flow-cloud into wei-epic-aws

avoid change wxflow

a2802aa

Merge branch 'wei-epic-aws' of github.com:NOAA-EPIC/global-workflow-c…

357ed59

…loud into wei-epic-aws

remove echo

d6b0f71

sync and clean up

a88ca29

sync on hera and test

80eced9

weihuang-jedi marked this pull request as ready for review June 14, 2024 20:35

aerorahul requested changes Jun 14, 2024

View reviewed changes

save before rebase

9e26c32

aerorahul closed this Jun 25, 2024

weihuang-jedi deleted the wei-epic-aws branch August 9, 2024 14:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement global-workflow on AWS #2549

Implement global-workflow on AWS #2549

weihuang-jedi commented Apr 27, 2024 •

edited

Loading

weihuang-jedi commented Apr 27, 2024

aerorahul left a comment

aerorahul Jun 14, 2024

aerorahul Jun 14, 2024

aerorahul Jun 14, 2024

aerorahul Jun 14, 2024

aerorahul Jun 14, 2024

aerorahul Jun 14, 2024

aerorahul Jun 14, 2024

aerorahul Jun 14, 2024

aerorahul Jun 14, 2024

aerorahul Jun 14, 2024

HenryRWinterbottom commented Jun 14, 2024

DavidHuber-NOAA commented Jun 25, 2024

aerorahul commented Jun 25, 2024

		pw_csp = os.environ.get('PW_CSP')
		if pw_csp not in ['aws', 'azure', 'google']:

Implement global-workflow on AWS #2549

Implement global-workflow on AWS #2549

Conversation

weihuang-jedi commented Apr 27, 2024 • edited Loading

Description

Type of change

Change characteristics

How has this been tested?

Checklist

weihuang-jedi commented Apr 27, 2024

aerorahul left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HenryRWinterbottom commented Jun 14, 2024

DavidHuber-NOAA commented Jun 25, 2024

aerorahul commented Jun 25, 2024

weihuang-jedi commented Apr 27, 2024 •

edited

Loading