CI Feature to detect stalled experiments #2049

TerrenceMcGuinness-NOAA · 2023-11-09T20:21:57Z

Description

A helper python script rocoto_statcount.py is added in $HOMEgfs/ci/scrpts/utils that checks to see if an experiment is "stalled". This is done by running rocotorun and checking within all the jobs and if none are RUNNING, SUBMITTING, or QUEUED then the experiment is STALLED.

Resolves #2008

Type of change

New feature (adds functionality)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO

How has this been tested?

Ran in CI CRON development space and set up a case that had a missing file dependency and it was appropriately flagged.

Checklist

Any dependent changes have been merged and published
My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings
New and existing tests pass with my changes
I have made corresponding changes to the documentation if necessary

quotes from bash linter

jobs should not be plurl

ci/scripts/run_ci.sh

fixed some grammar in description

ci/scripts/run_ci.sh

-    echo "Running: ${rocotorun} -v 10 -w ${xml} -d ${db}"
-    "${rocotorun}" -v 10 -w "${xml}" -d "${db}"
+    set +e
+    ${ROOT_DIR}/ci/scripts/utils/rocoto_statcount.py -d "${db}" -w "${xml}" --check_stalled


fixed double quotes for bash lint

aerorahul

looks ok, but I spotted some minor typos.
See suggestions.
And one comment on something that probably is incorrect.

aerorahul · 2023-11-09T20:26:28Z

ci/scripts/check_ci.sh

@@ -131,7 +131,7 @@ for pr in ${pr_list}; do
      "${GH}" pr edit --repo "${REPO_URL}" "${pr}" --remove-label "CI-${MACHINE_ID^}-Running" --add-label "CI-${MACHINE_ID^}-Failed"
      error_logs=$("${rocotostat}" -d "${db}" -w "${xml}" | grep -E 'FAIL|DEAD' | awk '{print "-c", $1, "-t", $2}' | xargs "${rocotocheck}" -d "${db}" -w "${xml}" | grep join | awk '{print $2}') || true
      {
-       echo "Experiment ${pslot} Terminated: *** FAILED ***"
+       echo "Experiment ${pslot} Terminated: *** FAILED *** on ${MACHIND_ID^}"


Small typo

Suggested change

echo "Experiment ${pslot} Terminated: *** FAILED *** on ${MACHIND_ID^}"

echo "Experiment ${pslot} Terminated: *** FAILED *** on ${MACHINE_ID^}"

aerorahul · 2023-11-09T20:28:11Z

ci/scripts/check_ci.sh

+      echo "Experiment ${pslot} **SUCCESS** ${DATE +%Y%m%d} on ${MACHINE_ID^}" >> "${output_ci_single}"
+      echo "Experiment ${pslot} **SUCCESS** at ${DATE +%Y%m%d} on ${MACHIND_ID^}" >> "${output_ci}"


This cannot possibly work ${DATE +%Y%m%d}.

aerorahul · 2023-11-09T20:28:54Z

ci/scripts/run_ci.sh

+      output_ci="${pr_dir}/output_runtime_single.log"
+      {
+        echo "${pslot} has *** STALLED **** on ${MACHINE_ID^}"
+        echo "A job in expermint ${pslot} in ${pslot_dir}"


Suggested change

echo "A job in expermint ${pslot} in ${pslot_dir}"

echo "A job in experiment ${pslot} in ${pslot_dir}"

ci/scripts/utils/rocoto_statcount.py

aerorahul

some suggestions for rocoto_statcount.py

ci/scripts/utils/rocoto_statcount.py

aerorahul · 2023-11-09T20:47:48Z

ci/scripts/utils/rocoto_statcount.py

+    xml_file_path = os.path.abspath(args.w)
+    db_file_path = os.path.abspath(args.d)
+
+    rocotostat_all = which("rocotostat")


Why do this again? You already have rocotostat. You can use that any number of times with different arguments.

@aerorahul I have not been able to find a way to use add_default_args to "update" the argument list of an Executable Object. Can you fine an example, could not find such a use case in the repo.

Could not find a way to "change" arguments once the are set on the Executable object.

aerorahul · 2023-11-09T20:48:25Z

ci/scripts/utils/rocoto_statcount.py

+    rocotostat.add_default_arg(['-w',xml_file_path,'-d',db_file_path,'-s'])
+    rocotostat_all.add_default_arg(['-w',xml_file_path,'-d',db_file_path,'-a'])
+
+    rocotostat_output = rocotostat(output=str)
+    rocotostat_output = rocotostat_output.splitlines()[1:]
+    rocotostat_output = [line.split()[0:2] for line in rocotostat_output]
+
+    rocotostat_output_all = rocotostat_all(output=str)
+    rocotostat_output_all = rocotostat_output_all.splitlines()[1:]
+    rocotostat_output_all = [line.split()[0:4] for line in rocotostat_output_all]
+    rocotostat_output_all = [line for line in rocotostat_output_all if len(line) != 1]


I can show you how to reduce this and simplify it considerably

Sure that would be great, you can jot down pseudo code it you like

aerorahul · 2023-11-09T20:50:02Z

ci/scripts/utils/rocoto_statcount.py

+
+    if args.check_stalled:
+        if rocoto_state != 'Done':
+            rocoto_run = which("rocotorun")


Having seen this, I think we should have a RocotoCommands class and initialize it with everything from rocoto you need; e.g. rocotostat, rocotorun, etc. While you are at it, also provide the xml and database to the class object rather than have to do this over and over.

ci/scripts/utils/rocoto_statcount.py

ci/scripts/check_ci.sh

@@ -152,8 +152,8 @@
      rm -f "${output_ci_single}"
      # echo "\`\`\`" > "${output_ci_single}"
      DATE=$(date)
-      echo "Experiment ${pslot} **SUCCESS** ${DATE}" >> "${output_ci_single}"
-      echo "Experiment ${pslot} **SUCCESS** at ${DATE}" >> "${output_ci}"
+      echo "Experiment ${pslot} **SUCCESS** $(date +'%A %b %d, %Y') on ${MACHINE_ID^}" >> "${output_ci_single}"


ci/scripts/check_ci.sh

-      echo "Experiment ${pslot} **SUCCESS** ${DATE}" >> "${output_ci_single}"
-      echo "Experiment ${pslot} **SUCCESS** at ${DATE}" >> "${output_ci}"
+      echo "Experiment ${pslot} **SUCCESS** $(date +'%A %b %d, %Y') on ${MACHINE_ID^}" >> "${output_ci_single}"
+      echo "Experiment ${pslot} **SUCCESS** at $(date +'%A %b %d, %Y') on ${MACHINE_ID^}" >> "${output_ci}"


ci/scripts/check_ci.sh

update to more robust logic Co-authored-by: Walter Kolczynski - NOAA <[email protected]>

added correct exception when command is not found Co-authored-by: Rahul Mahajan <[email protected]>

added import for correct exception Co-authored-by: Rahul Mahajan <[email protected]>

… UNKNOWN as a final state

… alternative argument list to rocotostat

or true for echo date commands

syntax fix on quote in rocotorun command line

removed old switch for check stalled that has been depricated

trying to force refresh

stabs at solving double quote requirements for shellnoms

double quoting flags wont stop even though the are correct

still wrestling with double quote complaints from shellnormns even thought they are correct in the script and trying that ever the F&*@

TerrenceMcGuinness-NOAA

Done with my review

TerrenceMcGuinness-NOAA · 2023-11-14T21:20:29Z

ci/scripts/utils/rocoto_statcount.py

+    xml_file_path = os.path.abspath(args.w)
+    db_file_path = os.path.abspath(args.d)
+
+    rocotostat_all = which("rocotostat")


Could not find a way to "change" arguments once the are set on the Executable object.

ci/scripts/run_ci.sh

WalterKolczynski-NOAA · 2023-11-15T20:23:57Z

ci/scripts/utils/rocoto_statcount.py

+    description = """
+        Using rocotostat to get the status of all jobs this scripts
+        determines rocoto_state: if all cycles are done, then rocoto_state is Done.
+        Assuming rocotorun had just been run, and the rocoto_state is not Done, then
+        rocoto_state is Stalled if there are no jobs that are RUNNING, SUBMITTING, or QUEUED.
+        """


This seems more like a description for the whole script or rocoto_statcount than this function.

WalterKolczynski-NOAA · 2023-11-17T21:04:07Z

@TerryMcGuinness-NOAA still waiting on you to make the suggestions I recommended

Co-authored-by: Walter Kolczynski - NOAA <[email protected]>

TerryMcGuinness-NOAA and others added 11 commits November 8, 2023 17:05

started to add python rocoto stat

fb871ab

first steps with rocoto stat script

b865bf2

finished working version of rocoto_statcount

1e59bef

replaced explicit decleration of stat dict with loop in rocoto statcount

dd4c735

fixed end state mixup of varialble names

f31ec82

more slight improvments on user GitHub messaging outputs

aad09fd

added failed state updates with stalled in run ci

d45017a

Update run_ci.sh

b170ca7

quotes from bash linter

Update run_ci.sh

ea4da7d

jobs should not be plurl

bug updates after testing depedancies catch

d529dbb

fixed conflict with run_ci

12c26e8

TerrenceMcGuinness-NOAA added feature New feature or request CI/CD Issue related to CI/CD labels Nov 9, 2023

TerrenceMcGuinness-NOAA requested review from aerorahul and WalterKolczynski-NOAA November 9, 2023 20:21

TerrenceMcGuinness-NOAA self-assigned this Nov 9, 2023

github-advanced-security bot found potential problems Nov 9, 2023

View reviewed changes

ci/scripts/run_ci.sh Fixed Show fixed Hide fixed

Update rocoto_statcount.py

5ae0d72

fixed some grammar in description

github-advanced-security bot found potential problems Nov 9, 2023

View reviewed changes

Update run_ci.sh

5b98e01

fixed double quotes for bash lint

aerorahul reviewed Nov 9, 2023

View reviewed changes

WalterKolczynski-NOAA requested changes Nov 9, 2023

View reviewed changes

ci/scripts/utils/rocoto_statcount.py Outdated Show resolved Hide resolved

ci/scripts/utils/rocoto_statcount.py Outdated Show resolved Hide resolved

aerorahul reviewed Nov 9, 2023

View reviewed changes

TerrenceMcGuinness-NOAA marked this pull request as draft November 13, 2023 15:38

spelling, typos, and brain fart on {DATE +}

b48f519

github-advanced-security bot found potential problems Nov 13, 2023

View reviewed changes

ci/scripts/check_ci.sh Fixed Show fixed Hide fixed

ci/scripts/check_ci.sh Fixed Show fixed Hide fixed

TerrenceMcGuinness-NOAA and others added 3 commits November 13, 2023 12:59

Update ci/scripts/utils/rocoto_statcount.py

02bd37e

update to more robust logic Co-authored-by: Walter Kolczynski - NOAA <[email protected]>

Update ci/scripts/utils/rocoto_statcount.py

ba7f6d9

added correct exception when command is not found Co-authored-by: Rahul Mahajan <[email protected]>

Update ci/scripts/utils/rocoto_statcount.py

729002c

added import for correct exception Co-authored-by: Rahul Mahajan <[email protected]>

TerryMcGuinness-NOAA and others added 10 commits November 13, 2023 19:58

put Walters test back for no RUNNING, SUBMITTING, or QUEUED and added…

b9a2d08

… UNKNOWN as a final state

added back which method to get new and sperate Execute Object to have…

1ca1097

… alternative argument list to rocotostat

Update check_ci.sh

ae0ca41

or true for echo date commands

Update run_ci.sh

fd8c33b

syntax fix on quote in rocotorun command line

Update run_ci.sh

8146364

removed old switch for check stalled that has been depricated

removed DATE as date command

688613f

white spaces from lint

e74567f

pynorm indent on dict

1654e52

more white space related pynorm stuff

919b0c2

hopefully last white space related pynorm stuff

1bfe74a

TerrenceMcGuinness-NOAA marked this pull request as ready for review November 13, 2023 21:09

TerrenceMcGuinness-NOAA and others added 7 commits November 13, 2023 16:11

Update run_ci.sh

ffd92cf

trying to force refresh

Update run_ci.sh

084719b

stabs at solving double quote requirements for shellnoms

Update run_ci.sh

14e0df1

double quoting flags wont stop even though the are correct

Update run_ci.sh

18557c0

still wrestling with double quote complaints from shellnormns even thought they are correct in the script and trying that ever the F&*@

Merge branch 'NOAA-EMC:develop' into feature/detect_dependency

9639ae3

Merge branch 'NOAA-EMC:develop' into feature/detect_dependency

52bf3bb

updated discription for rocoto_statcount

451b55c

TerrenceMcGuinness-NOAA commented Nov 14, 2023

View reviewed changes

TerrenceMcGuinness-NOAA requested review from aerorahul and WalterKolczynski-NOAA November 14, 2023 21:20

WalterKolczynski-NOAA reviewed Nov 15, 2023

View reviewed changes

Merge branch 'NOAA-EMC:develop' into feature/detect_dependency

cf3b262

aerorahul requested a review from WalterKolczynski-NOAA November 17, 2023 20:55

TerrenceMcGuinness-NOAA marked this pull request as draft November 28, 2023 23:05

TerrenceMcGuinness-NOAA and others added 2 commits November 28, 2023 18:06

Update ci/scripts/run_ci.sh

338831c

Co-authored-by: Walter Kolczynski - NOAA <[email protected]>

Update ci/scripts/run_ci.sh

a839b4b

Co-authored-by: Walter Kolczynski - NOAA <[email protected]>

TerrenceMcGuinness-NOAA closed this Nov 28, 2023

TerrenceMcGuinness-NOAA deleted the feature/detect_dependency branch July 12, 2024 15:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI Feature to detect stalled experiments #2049

CI Feature to detect stalled experiments #2049

TerrenceMcGuinness-NOAA commented Nov 9, 2023 •

edited

Loading

aerorahul left a comment

aerorahul Nov 9, 2023

aerorahul Nov 9, 2023

aerorahul Nov 9, 2023

aerorahul left a comment

aerorahul Nov 9, 2023 •

edited by TerrenceMcGuinness-NOAA

Loading

TerrenceMcGuinness-NOAA Nov 14, 2023

aerorahul Nov 9, 2023

TerrenceMcGuinness-NOAA Nov 14, 2023

aerorahul Nov 9, 2023

TerrenceMcGuinness-NOAA left a comment

TerrenceMcGuinness-NOAA Nov 14, 2023

WalterKolczynski-NOAA Nov 15, 2023

WalterKolczynski-NOAA commented Nov 17, 2023

	echo "Experiment ${pslot} Terminated: * FAILED * on ${MACHIND_ID^}"
	echo "Experiment ${pslot} Terminated: * FAILED * on ${MACHINE_ID^}"

		echo "Experiment ${pslot} SUCCESS ${DATE +%Y%m%d} on ${MACHINE_ID^}" >> "${output_ci_single}"
		echo "Experiment ${pslot} SUCCESS at ${DATE +%Y%m%d} on ${MACHIND_ID^}" >> "${output_ci}"

	echo "A job in expermint ${pslot} in ${pslot_dir}"
	echo "A job in experiment ${pslot} in ${pslot_dir}"

CI Feature to detect stalled experiments #2049

CI Feature to detect stalled experiments #2049

Conversation

TerrenceMcGuinness-NOAA commented Nov 9, 2023 • edited Loading

Description

Type of change

Change characteristics

How has this been tested?

Checklist

aerorahul left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aerorahul left a comment

Choose a reason for hiding this comment

aerorahul Nov 9, 2023 • edited by TerrenceMcGuinness-NOAA Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TerrenceMcGuinness-NOAA left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WalterKolczynski-NOAA commented Nov 17, 2023

TerrenceMcGuinness-NOAA commented Nov 9, 2023 •

edited

Loading

aerorahul Nov 9, 2023 •

edited by TerrenceMcGuinness-NOAA

Loading