refactor: collect step artifacts immediately after step finish #718

miltolstoy · 2022-05-09T09:36:49Z

This change was requested for conditional steps artifacts collections easier development.
Relates #717

Logic before:

Whole artifacts list preprocessing
Steps execution
Whole artifacts list collection

Logic now:

Whole artifacts list preprocessing (same)
For each step:
2.1. Step execution
2.2. Step's artifacts collection

Log before:

5. Executing build steps
 |   5.1.  [ 1/4 ] Step with artifacts
 |      |   ==> Adding file /home/alelyak/Desktop/universum_test/artifacts/Step_with_artifacts_log.txt to artifacts...
 |      |   ==> Execution log is redirected to file
 |      |   $ /usr/bin/touch artifact1
 |      └ [Success]
 |
 |   5.2.  [ 2/4 ] Step with no artifacts
 |      |   ==> Adding file /home/alelyak/Desktop/universum_test/artifacts/Step_with_no_artifacts_log.txt to artifacts...
 |      |   ==> Execution log is redirected to file
 |      |   ==> No 'command' found. Nothing to execute
 |      └ [Success]
 |
 |   5.3.  [ 3/4 ] Failed step with artifacts
 |      |   ==> Adding file /home/alelyak/Desktop/universum_test/artifacts/Failed_step_with_artifacts_log.txt to artifacts...
 |      |   ==> Execution log is redirected to file
 |      |   $ /usr/bin/bash -c 'touch artifact4; exit 1'
 |      |   Error: Module sh got exit code 1
 |      └ [Failed]
 |
 |   Critical step failed. All further configurations will be skipped
 |   5.4. [ 4/4 ] 'Skipped step' skipped because of critical step failure
 └ [Failed]

6. Collecting artifacts
 |   6.1. Collecting 'artifact4'
 |      |   ==> Processing '/home/alelyak/Desktop/universum_test/temp/artifact4'
 |      └ [Success]
 |
 |   6.2. Collecting 'artifact1'
 |      |   ==> Processing '/home/alelyak/Desktop/universum_test/temp/artifact1'
 |      └ [Success]
 |
 └ [Success]

7. Reporting build result
 |   ==> Here is the summarized build result:
 |   ==> 5. Executing build steps
 |   ==>   5.1.  [ 1/4 ] Step with artifacts - Success
 |   ==>   5.2.  [ 2/4 ] Step with no artifacts - Success
 |   ==>   5.3.  [ 3/4 ] Failed step with artifacts - Failed
 |   ==>   5.4.  [ 4/4 ] 'Skipped step' - Skipped
 |   ==> 6. Collecting artifacts
 |   ==>   6.1. Collecting 'artifact4' - Success
 |   ==>   6.2. Collecting 'artifact1' - Success
 |   ==> Nowhere to report. Skipping...
 └ [Success]

Log now:

5. Executing build steps
 |   5.1.  [ 1/4 ] Step with artifacts
 |      |   ==> Adding file /home/alelyak/Desktop/universum_test/artifacts/Step_with_artifacts_log.txt to artifacts...
 |      |   ==> Execution log is redirected to file
 |      |   $ /usr/bin/touch artifact1
 |      └ [Success]
 |
 |   5.2. Collecting artifacts for the 'Step with artifacts' step
 |      |   ==> Processing '/home/alelyak/Desktop/universum_test/temp/artifact1'
 |      └ [Success]
 |
 |   5.3.  [ 2/4 ] Step with no artifacts
 |      |   ==> Adding file /home/alelyak/Desktop/universum_test/artifacts/Step_with_no_artifacts_log.txt to artifacts...
 |      |   ==> Execution log is redirected to file
 |      |   ==> No 'command' found. Nothing to execute
 |      └ [Success]
 |
 |   5.4. Collecting artifacts for the 'Step with no artifacts' step
 |      |   ==> Processing '/home/alelyak/Desktop/universum_test/temp/artifact4'
 |      |   Error: No artifacts found!
 |      |   Possible reasons of this error:
 |      |    * Artifact was not created while building the project due to some internal errors
 |      |    * Artifact path was not specified correctly in 'configs.py'
 |      └ [Failed]
 |
 |   5.5.  [ 3/4 ] Failed step with artifacts
 |      |   ==> Adding file /home/alelyak/Desktop/universum_test/artifacts/Failed_step_with_artifacts_log.txt to artifacts...
 |      |   ==> Execution log is redirected to file
 |      |   $ /usr/bin/bash -c 'touch artifact4; exit 1'
 |      |   Error: Module sh got exit code 1
 |      └ [Failed]
 |
 |   5.6. Collecting artifacts for the 'Failed step with artifacts' step
 |      |   ==> Processing '/home/alelyak/Desktop/universum_test/temp/artifact4'
 |      └ [Success]
 |
 |   Critical step failed. All further configurations will be skipped
 |   5.7. [ 4/4 ] 'Skipped step' skipped because of critical step failure
 └ [Failed]

6. Reporting build result
 |   ==> Here is the summarized build result:
 |   ==> 5. Executing build steps
 |   ==>   5.1.  [ 1/4 ] Step with artifacts - Success
 |   ==>   5.2. Collecting artifacts for the 'Step with artifacts' step - Success
 |   ==>   5.3.  [ 2/4 ] Step with no artifacts - Success
 |   ==>   5.4. Collecting artifacts for the 'Step with no artifacts' step - Failed
 |   ==>   5.5.  [ 3/4 ] Failed step with artifacts - Failed
 |   ==>   5.6. Collecting artifacts for the 'Failed step with artifacts' step - Success
 |   ==>   5.7.  [ 4/4 ] 'Skipped step' - Skipped
 |   ==> Nowhere to report. Skipping...
 └ [Success]

Background steps example:

5. Executing build steps
 |   5.1.  [ 1/2 ] Background step with artifacts
 |      |   ==> Adding file /home/alelyak/Desktop/universum_test/artifacts/Background_step_with_artifacts_log.txt to artifacts...
 |      |   ==> Execution log is redirected to file
 |      |   $ /usr/bin/bash -c 'sleep 2; touch artifact3'
 |      |   ==> This step is marked to be executed in background
 |      └ [Success]
 |
 |   5.2.  [ 2/2 ] Background step with no artifacts
 |      |   ==> Adding file /home/alelyak/Desktop/universum_test/artifacts/Background_step_with_no_artifacts_log.txt to artifacts...
 |      |   ==> Execution log is redirected to file
 |      |   $ /usr/bin/bash -c 'sleep 1'
 |      |   ==> This step is marked to be executed in background
 |      └ [Success]
 |
 |   5.3. Reporting background steps
 |      |   5.3.1. Waiting for background step 'Background step with artifacts' to finish...
 |      |        └ [Success]
 |      |
 |      |   5.3.2. Collecting artifacts for the 'Background step with artifacts' step
 |      |        |   ==> Processing '/home/alelyak/Desktop/universum_test/temp/artifact3'
 |      |        └ [Success]
 |      |
 |      |   5.3.3. Waiting for background step 'Background step with no artifacts' to finish...
 |      |        └ [Success]
 |      |
 |      |   5.3.4. Collecting artifacts for the 'Background step with no artifacts' step
 |      |        |   ==> Processing '/home/alelyak/Desktop/universum_test/temp/artifact1'
 |      |        |   Error: No artifacts found!
 |      |        |   Possible reasons of this error:
 |      |        |    * Artifact was not created while building the project due to some internal errors
 |      |        |    * Artifact path was not specified correctly in 'configs.py'
 |      |        └ [Failed]
 |      |
 |      |   ==> All ongoing background steps completed
 |      └ [Success]
 |
 └ [Success]

6. Reporting build result
 |   ==> Here is the summarized build result:
 |   ==> 5. Executing build steps
 |   ==>   5.1.  [ 1/2 ] Background step with artifacts - Success
 |   ==>   5.2.  [ 2/2 ] Background step with no artifacts - Success
 |   ==>   5.3. Reporting background steps
 |   ==>     5.3.1. Waiting for background step 'Background step with artifacts' to finish... - Success
 |   ==>     5.3.2. Collecting artifacts for the 'Background step with artifacts' step - Success
 |   ==>     5.3.3. Waiting for background step 'Background step with no artifacts' to finish... - Success
 |   ==>     5.3.4. Collecting artifacts for the 'Background step with no artifacts' step - Failed
 |   ==> Nowhere to report. Skipping...
 └ [Success]

k-dovgan · 2022-05-12T11:57:27Z

Can it be transformed a little so that collecting artifacts after the step is displayed as a separate step or a substep?

miltolstoy · 2022-05-16T06:26:41Z

Can it be transformed a little so that collecting artifacts after the step is displayed as a separate step or a substep?

Let's discuss and choose the one.

Option 1: Collect artifacts at the end of the step (current one)

 |   5.1.  [ 1/2 ] Step with artifacts
 |      |   ==> Adding file /home/alelyak/Desktop/universum_test/artifacts/Step_with_artifacts_log.txt to artifacts...
 |      |   ==> Execution log is redirected to file
 |      |   $ /usr/bin/touch artifact1
 |      |   ==> Processing '/home/alelyak/Desktop/universum_test/temp/artifact1'
 |      └ [Success]
 |
 |   5.2.  [ 2/2 ] Step with no artifacts
 |      |   ==> Adding file /home/alelyak/Desktop/universum_test/artifacts/Step_with_no_artifacts_log.txt to artifacts...
 |      |   ==> Execution log is redirected to file
 |      |   ==> No 'command' found. Nothing to execute
 |      |   ==> Processing '/home/alelyak/Desktop/universum_test/temp/artifact4'
 |      |   Error: No artifacts found!
 |      |   Possible reasons of this error:
 |      |    * Artifact was not created while building the project due to some internal errors
 |      |    * Artifact path was not specified correctly in 'configs.py'
 |      └ [Failed]

Option 2: Collect artifacts in the substep

 |   5.1.  [ 1/2 ] Step with artifacts
 |      |   ==> Adding file /home/alelyak/Desktop/universum_test/artifacts/Step_with_artifacts_log.txt to artifacts...
 |      |   ==> Execution log is redirected to file
 |      |   $ /usr/bin/touch artifact1
 |      |   5.1.1. Collecting 'artifact1'
 |      |        |   ==> Processing '/home/alelyak/Desktop/universum_test/temp/artifact1'
 |      |        └ [Success]
 |      |
 |      └ [Success]
 |
 |   5.2.  [ 2/2 ] Step with no artifacts
 |      |   ==> Adding file /home/alelyak/Desktop/universum_test/artifacts/Step_with_no_artifacts_log.txt to artifacts...
 |      |   ==> Execution log is redirected to file
 |      |   ==> No 'command' found. Nothing to execute
 |      |   5.2.1. Collecting 'artifact4'
 |      |        |   ==> Processing '/home/alelyak/Desktop/universum_test/temp/artifact4'
 |      |        |   Error: No artifacts found!
 |      |        |   Possible reasons of this error:
 |      |        |    * Artifact was not created while building the project due to some internal errors
 |      |        |    * Artifact path was not specified correctly in 'configs.py'
 |      |        └ [Failed]
 |      |
 |      └ [Success]

Option 3: Collect artifacts in the separate step
Can be also modified to count "artifact collection" steps like regular ones in steps numeration

 |   5.1.  [ 1/2 ] Step with artifacts
 |      |   ==> Adding file /home/alelyak/Desktop/universum_test/artifacts/Step_with_artifacts_log.txt to artifacts...
 |      |   ==> Execution log is redirected to file
 |      |   $ /usr/bin/touch artifact1
 |      └ [Success]
 |
 |   5.2. Collecting 'artifact1'
 |      |   ==> Processing '/home/alelyak/Desktop/universum_test/temp/artifact1'
 |      └ [Success]
 |
 |   5.3.  [ 2/2 ] Step with no artifacts
 |      |   ==> Adding file /home/alelyak/Desktop/universum_test/artifacts/Step_with_no_artifacts_log.txt to artifacts...
 |      |   ==> Execution log is redirected to file
 |      |   ==> No 'command' found. Nothing to execute
 |      └ [Success]
 |
 |   5.4. Collecting 'artifact4'
 |      |   ==> Processing '/home/alelyak/Desktop/universum_test/temp/artifact4'
 |      |   Error: No artifacts found!
 |      |   Possible reasons of this error:
 |      |    * Artifact was not created while building the project due to some internal errors
 |      |    * Artifact path was not specified correctly in 'configs.py'
 |      └ [Failed]

k-dovgan · 2022-05-16T10:44:32Z

I myself prefer the option 3. It seems the most readable to me. But this way step numeration may be more confusing to the user.

miltolstoy · 2022-05-18T09:04:50Z

PR description was updated according to changes in the last commit.

universum/modules/artifact_collector.py

universum/modules/launcher.py

i-keliukh · 2022-05-21T12:54:26Z

Please also update the documentation to reflect the changes.

universum/modules/artifact_collector.py

universum/modules/launcher.py

universum/modules/structure_handler.py

k-dovgan · 2022-06-28T11:22:10Z

universum/modules/structure_handler.py

+                    try:
+                        step_process = self.run_in_block(self.execute_one_step, step_name, False,
+                                                         item, step_executor, obj_a.critical)
+                        if not item.background:


I also kinda don't like the

if not item.background: self._run_artifact_collection_step(item.name, step_process.collect_artifacts) <one line> if not item.background: self._run_artifact_collection_step(item.name, e.step_process.collect_artifacts)

part. These lines seem a little too similar. Are we sure this cannot be fixed somehow?

a-osipov-toxa · 2022-07-05T11:28:19Z

Please note that this is breaking change. Some configs expect that artifacts will be collected at the end of all steps.
Probably, Universum release must be tested with these projects before delivery on build servers
Thanks

miltolstoy · 2022-07-26T04:26:25Z

Log examples in the PR description were extended.

universum/modules/structure_handler.py

…tifacts_after_step

refactor: collect step artifacts immediately after step finish

445cd70

miltolstoy requested review from k-dovgan and i-keliukh May 9, 2022 10:42

move artifact collection to a separate step

78cb7b6

i-keliukh reviewed May 20, 2022

View reviewed changes

universum/modules/artifact_collector.py Outdated Show resolved Hide resolved

i-keliukh reviewed May 20, 2022

View reviewed changes

universum/modules/artifact_collector.py Show resolved Hide resolved

i-keliukh reviewed May 20, 2022

View reviewed changes

universum/modules/launcher.py Outdated Show resolved Hide resolved

miltolstoy added 3 commits May 23, 2022 08:59

fix pylint issue & test

d1587ac

move step's artifact collection logic to StructureHandler

ab4c805

collect artifacts after failed step too

6e8809b

i-keliukh reviewed Jun 27, 2022

View reviewed changes

universum/modules/artifact_collector.py Outdated Show resolved Hide resolved

i-keliukh reviewed Jun 27, 2022

View reviewed changes

universum/modules/launcher.py Outdated Show resolved Hide resolved

i-keliukh reviewed Jun 27, 2022

View reviewed changes

universum/modules/structure_handler.py Outdated Show resolved Hide resolved

i-keliukh reviewed Jun 27, 2022

View reviewed changes

universum/modules/structure_handler.py Outdated Show resolved Hide resolved

review comments minor fixes

40c6317

k-dovgan reviewed Jun 28, 2022

View reviewed changes

miltolstoy added 2 commits July 25, 2022 05:45

Merge branch 'collect_artifacts_after_step'

34bd5bc

re-apply changes on new master

7b2476d

fix CI issues

d043087

miltolstoy requested review from i-keliukh and k-dovgan July 26, 2022 05:53

i-keliukh reviewed Jul 26, 2022

View reviewed changes

universum/modules/structure_handler.py Outdated Show resolved Hide resolved

i-keliukh reviewed Jul 26, 2022

View reviewed changes

universum/modules/structure_handler.py Outdated Show resolved Hide resolved

i-keliukh reviewed Jul 26, 2022

View reviewed changes

universum/modules/structure_handler.py Outdated Show resolved Hide resolved

review comments partial fix

9755dc8

miltolstoy mentioned this pull request Aug 1, 2022

refactor(launcher): add RunningStep interface #728

Merged

miltolstoy added 4 commits August 4, 2022 10:37

Merge branch 'master' into collect_artifacts_after_step

c16cbfa

update Step class docstring

d42b892

restore artifacts reporting

73057cc

fix static analysis errors

4c19eb7

miltolstoy requested a review from i-keliukh August 5, 2022 06:18

i-keliukh reviewed Aug 5, 2022

View reviewed changes

universum/modules/structure_handler.py Outdated Show resolved Hide resolved

i-keliukh reviewed Aug 5, 2022

View reviewed changes

universum/modules/structure_handler.py Show resolved Hide resolved

i-keliukh reviewed Aug 5, 2022

View reviewed changes

universum/modules/structure_handler.py Outdated Show resolved Hide resolved

i-keliukh reviewed Aug 5, 2022

View reviewed changes

universum/modules/structure_handler.py Show resolved Hide resolved

miltolstoy added 4 commits September 14, 2022 15:51

Merge branch 'master' into collect_artifacts_after_step

8b6f2a3

create artifacts colletion step only if artifacts are present

6711770

add more annotations

273cd0f

count also

24d8d6b

miltolstoy requested a review from i-keliukh September 15, 2022 06:17

miltolstoy added 3 commits September 20, 2022 09:48

remove mypy error ignoring

c680247

Merge branch 'master' of github.com:Samsung/Universum into collect_ar…

2a794d7

…tifacts_after_step

fix pylint issue

6cdec59

i-keliukh approved these changes Sep 20, 2022

View reviewed changes

miltolstoy merged commit b7a66e9 into master Sep 20, 2022

miltolstoy deleted the collect_artifacts_after_step branch September 20, 2022 14:03

This was referenced Oct 5, 2022

Add reference to step for each collected artifact #295

Open

Skip artifacts collecting for the step that was skipped #168

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: collect step artifacts immediately after step finish #718

refactor: collect step artifacts immediately after step finish #718

miltolstoy commented May 9, 2022 •

edited

Loading

k-dovgan commented May 12, 2022

miltolstoy commented May 16, 2022

k-dovgan commented May 16, 2022

miltolstoy commented May 18, 2022

i-keliukh commented May 21, 2022

k-dovgan Jun 28, 2022 •

edited

Loading

a-osipov-toxa commented Jul 5, 2022

miltolstoy commented Jul 26, 2022

refactor: collect step artifacts immediately after step finish #718

refactor: collect step artifacts immediately after step finish #718

Conversation

miltolstoy commented May 9, 2022 • edited Loading

k-dovgan commented May 12, 2022

miltolstoy commented May 16, 2022

k-dovgan commented May 16, 2022

miltolstoy commented May 18, 2022

i-keliukh commented May 21, 2022

k-dovgan Jun 28, 2022 • edited Loading

Choose a reason for hiding this comment

a-osipov-toxa commented Jul 5, 2022

miltolstoy commented Jul 26, 2022

miltolstoy commented May 9, 2022 •

edited

Loading

k-dovgan Jun 28, 2022 •

edited

Loading