Dataflow Job Not Triggering in Kubeflow V2 Pipeline on Vertex AI

[MuhammadRehmanRabbani]

Hi,

I'm converting a Kubeflow pipeline from V1 to V2 and creating the V2 pipeline on Vertex AI. There is a part of the pipeline where I need to execute a Dataflow job. To run the Dataflow job, I'm using the following Dataflow component:
https://raw.githubusercontent.com/kubeflow/pipelines/1.7.0-rc.3/components/gcp/dataflow/launch_python/component.yaml
I compile the pipeline and create a run on Vertex AI. The Dataflow component task is triggered, but it does not create a job on Dataflow. However, this was running fine on V1 with the same component and input parameters.
There is no error in the logs of the component that can help resolve the issue. Following are the logs of the Dataflow component node. After the POST request, a job should be triggered on Dataflow.

2024-06-18 13:46:24.277 CEST
Tearing down training program.

2024-06-18 13:46:43.331 CEST
URL being requested: GET https://www.googleapis.com/discovery/v1/apis/dataflow/v1b3/rest

2024-06-18 13:46:43.331 CEST
Running without KFP context.

2024-06-18 13:46:43.332 CEST
Start KFP context with ID: 67422d6811ef96e5e6fad36405

2024-06-18 13:46:43.332 CEST
staging_location: gs://test-bucket/vertexai_test/staging/test/20240515/67422d6811ef96e5e6fad36405

2024-06-18 13:46:43.332 CEST
URL being requested: POST https://dataflow.googleapis.com/v1b3/projects/test/locations/europe-west3/templates:launch?gcsPath=gs%3A%2F%2Ftestzone%2Ftemplates%2Fraw&validateOnly=true&alt=json

2024-06-18 13:49:52.636 CEST
Finished tearing down training program.

2024-06-18 13:49:52.817 CEST
Job completed successfully.


How can I resolve this issue? Any help would be appreciated.





Thanks