[New Feature]: Scale up the SBG workflow

[LucaCinquini]

Processing:

• 1 day
• 1 month

Test: processing SBG end-to-end 1 hour

[drewm-jpl]

Do we have the CWL workflow and images in ECR available to be able to proceed with this?

[LucaCinquini]

MIke said he is working on a simplified CWL that only includes pre-processing and iso-fit.

[LucaCinquini]

Executed 10 concurrent SBG end-to-end workflows that contain pre-process + isofit on unity-venue-dev using these parameters:

data = {
"inputs": {
"cwl_workflow": "https://raw.githubusercontent.com/unity-sds/sbg-workflows/refs/heads/main/L1-to-L2-e2e.scale.cwl",
"cwl_args": "https://raw.githubusercontent.com/unity-sds/sbg-workflows/refs/heads/main/L1-to-L2-e2e.dev.scale.yml",
"request_memory": "64Gi",
"request_cpu": "32",
"request_storage": "100Gi",
"use_ecr": True
},
"outputs": {
"result": {
"transmissionMode": "reference"
}
}
}

All 10 workflows completed successfully in about 60 minutes, which is double the execution time of a single workflow standalone. This suggest there might be some bottlenecks in the infrastructure, perhaps networking on accessing the DS services? See graph below:

[mike-gangl]

the ASIPS folks did a breakdown of stage-in time, processing, and stage-out time. That might be helpful to see here.