[New Feature]: Scale up the EMIT workflow

[LucaCinquini]

Demonstrate scalable processing for the EMIT workflow: process 300 scenes.

Risk: scalability might be impacted by factors outside of SPS control which will require coordination, for example:

• DS data services
• NAT gateway

Test: processing of 1 EMIT scene

[LucaCinquini]

Processing of single workflow on unity-venue-dev took ~ 10 minutes.

[LucaCinquini]

Was able to submit 10 EMIT workflows at 60 secs interval, all completed with a time increase to about $15 minutes.

[LucaCinquini]

Using the same EMNIT Docker image from ECR did not seem to change the execution time very much. One of the jobs failed probably because the GitHub URL was temporarily unavailable:

[2024-11-18, 18:43:16 UTC] {pod_manager.py:472} INFO - [base] File "schema_salad/ref_resolver.py", line 1198, in validate_links
[2024-11-18, 18:43:16 UTC] {pod_manager.py:472} INFO - [base] schema_salad.exceptions.ValidationException: https://raw.githubusercontent.com/unity-sds/unity-sps-workflows/refs/heads/main/emit/workflow.cwl:80:1: checking field
[2024-11-18, 18:43:16 UTC] {pod_manager.py:472} INFO - [base] 'steps'
[2024-11-18, 18:43:16 UTC] {pod_manager.py:472} INFO - [base] https://raw.githubusercontent.com/unity-sds/unity-sps-workflows/refs/heads/main/emit/workflow.cwl:124:3: checking object
[2024-11-18, 18:43:16 UTC] {pod_manager.py:472} INFO - [base] 'https://raw.githubusercontent.com/unity-sds/unity-sps-workflows/refs/heads/main/emit/workflow.cwl#stage_out'
[2024-11-18, 18:43:16 UTC] {pod_manager.py:472} INFO - [base] https://raw.githubusercontent.com/unity-sds/unity-sps-workflows/refs/heads/main/emit/workflow.cwl:149:5: Field 'run'
[2024-11-18, 18:43:16 UTC] {pod_manager.py:472} INFO - [base] contains undefined
[2024-11-18, 18:43:16 UTC] {pod_manager.py:472} INFO - [base] reference to
[2024-11-18, 18:43:17 UTC] {pod_manager.py:490} INFO - [base] 'https://raw.githubusercontent.com/unity-sds/unity-sps-workflows/refs/heads/main/emit/stage_out.cwl'
[2024-11-18, 18:43:17 UTC] {pod_manager.py:623} INFO - Pod cwl-task-pod-j21e1np7 has phase Running
[2024-11-18, 18:43:19 UTC] {pod.py:984} INFO - Skipping deleting pod: cwl-task-pod-j21e1np7
[2024-11-18, 18:43:19 UTC] {taskinstance.py:3301} ERROR - Task failed with exception

Clearing the task and resubmitting the job resulted in success.