Python models don't wait for all-purpose-cluster to be fully ready

[mmenjivarST]

dbt-databricks/dbt/adapters/databricks/python_submissions.py

Line 404 in 38aefbf

def _wait_for_cluster_to_start(self) -> None:

Describe the bug

When executing python models in submission_method: all_purpose_cluster, the dbt-databricks connector is not recognizing when the cluster is still installing libraries. This produces python models to fail intermittently when libs are needed.

Steps To Reproduce

1. Use an All Purpose Cluster running 14.3LTS
2. Configure python models to run in all_purpose_cluster mode
3. Create a python model that use a library that requires installation and configure it in the cluster created in 1.
4. Execute the python model when the All Purpose Cluster is terminated

Expected behavior

Python models waiting until libraries configured in the cluster are installed

Screenshots and log output

Based on this function, it waits until the cluster is in a running state, but this doesn't mean that the cluster is ready to use as shown in the screenshot.

System information

The output of dbt --version:

1.8.3