DAB cannot be used to deploy both volume and artifact inside volume effectively

[Tello-Wharton]

Describe the issue

When a DAB defines both an artifact path inside a volume, and the volume as a resource, it cannot deploy. The advice given to deploy the volume and then add in the artifact feels substandard. Surely we should be able to use DABs to cold start what they define.

Configuration

resources:
  volumes:
    artifacts:
      name: artifacts
      schema_name: test_schema
      catalog_name: test_catalog
workspace: 
  host: <databricks-host>
  artifact_path: /Volumes/test_catalog/test_schema/artifacts/example_folder

Steps to reproduce the behavior

1. Run databricks bundle deploy
2. See error

Expected Behavior

The volume should be created first, and the bundle should deploy without issue

Actual Behavior

Error: volume /Volumes/test_catalog/test_schema/artifacts does not exist: Not Found
  at workspace.artifact_path
     resources.volumes.artifacts
  in databricks.yml:69:18
     databricks.yml:48:7

You are using a volume in your artifact_path that is managed by
this bundle but which has not been deployed yet. Please first deploy
the volume using 'bundle deploy' and then switch over to using it in
the artifact_path.

databricks deployment failed

OS and CLI version

OS: Ubuntu 22:04
CLI: 0.235.1-dev+0a36681b

Is this a regression?

I do not believe so

[shreyas-goenka]

Hi @Tello-Wharton, Thanks for reaching out!

DABs cannot create the UC Volume being used before uploading artifacts to workspace.artifact_path. This is a consequence of us using Terraform underneath the good to manage the CRUD lifecycle of resources.

We always sequence the terraform apply after uploading artifacts. Removing the error above would require us to apply both before and after uploading artifacts, which significantly complicates our deployment model.

We are planning to make fundamental improvements to this by eliminating the Terraform dependency entirely from DABs, but this patch is at least 3-6 months out.

I'll close this issue since it's a known limitation on our side but please feel free to reopen this if you have additional questions.

[Tello-Wharton]

@shreyas-goenka as a stop gap, would a viable solution be to add a flag which only deploys the volume? Or do you have a suggestion on how I can get my deployment bootstrapped that does not involve changing the actual source code of my config?

[shreyas-goenka]

@Tello-Wharton Could you please elaborate on what is blocking you? Why can't you:

1. Add the UC volume in your DAB
2. Deploy it
3. Then use it in artifact_path?

[Tello-Wharton]

@shreyas-goenka I provide the DAB expecting colleagues to use it to set up their dev environments, and use it to deploy QA environments in our CI/CD. Asking everyone to change config source code manually to get their environments bootstrapped (and having to do the same every time we create a qa environment) is quite annoying.

[shreyas-goenka]

Thanks, @Tello-Wharton. That's completely fair feedback. This is indeed a pain point that we should (and will) fix. A short-term fix is tricky because it's not just creating the volume but also managing all the associated states with the volume when you deploy one.

I'll keep this issue open, and we'll evaluate whether there are any feasible short-term fixes. Meanwhile, there is no workaround for this today unfortunately.

[Tello-Wharton]

@shreyas-goenka thanks for keeping it in mind