Added support for Databricks Apps in DABs #1928

andrewnester · 2024-11-21T16:26:28Z

Changes

Now it's possible to configure new app resource in bundle and point it to the custom source_code_path location where Databricks App code is defined.

On databricks bundle deploy DABs will create an app. All consecutive databricks bundle deploy execution will update an existing app if there are any updated

On databricks bundle run <my_app> DABs will execute app deployment. If the app is not started yet, it will start the app first.

Bundle configuration

bundle:
  name: apps

variables:
  my_job_id:
    description: "ID of job to run app"
    lookup:
      job: "My Job"
  databricks_name:
    description: "Name for app user"
  additional_flags:
    description: "Additional flags to run command app"
    default: ""
  my_app_config:
    type: complex
    description: "Configuration for my Databricks App"
    default:
      command:
        - flask
        - --app
        - hello
        - run
        - ${var.additional_flags}
      env:
        - name: DATABRICKS_NAME
          value: ${var.databricks_name}

resources:
  apps:
    my_app:
      name: "anester-app" # required and has to be unique
      description: "My App"
      source_code_path: ./app # required and points to location of app code
      config: ${var.my_app_config}
      resources:
        - name: "my-job"
          description: "A job for app to be able to run"
          job:
            id: ${var.my_job_id}
            permission: "CAN_MANAGE_RUN"
      permissions:
        - user_name: "[email protected]"
          level: "CAN_VIEW"
        - service_principal_name: "my_sp"
          level: "CAN_MANAGE"

targets:
  dev:
    variables:
      databricks_name: "Andrew (from dev)"
      additional_flags: --debug
  
  prod:
    variables:
      databricks_name: "Andrew (from prod)"

Execution

databricks bundle deploy -t dev
databricks bundle run my_app -t dev

If app is started

✓ Getting the status of the app my-app
✓ App is in RUNNING state
✓ Preparing source code for new app deployment.
✓ Deployment is pending
✓ Starting app with command: flask --app hello run --debug
✓ App started successfully
You can access the app at <app-url>

If app is not started

✓ Getting the status of the app my-app
✓ App is in UNAVAILABLE state
✓ Starting the app my-app
✓ App is starting...
....
✓ App is starting...
✓ App is started!
✓ Preparing source code for new app deployment.
✓ Downloading source code from /Workspace/Users/...
✓ Starting app with command: flask --app hello run --debug
✓ App started successfully
You can access the app at <app-url>

Tests

Added unit and config tests + manual test.

--- PASS: TestAccDeployBundleWithApp (404.59s)
PASS
coverage: 36.8% of statements in ./...
ok      github.com/databricks/cli/internal/bundle       405.035s        coverage: 36.8% of statements in ./...

bundle/internal/tf/schema/resource_app.go

## Changes Added support for bundle generate and bind for Apps ## Tests - [ ] Add E2E test

eng-dev-ecosystem-bot · 2024-12-18T13:34:59Z

Test Details: go/deco-tests/12394072048

bundle/apps/upload_config.go

github-actions · 2024-12-19T13:00:37Z

If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
go/deco-tests-run/cli

Inputs:

PR number: 1928
Commit SHA: 2f8df81159e2e20de06ae49d5aa290324100fd2c

Checks will be approved automatically on success.

pietern · 2024-12-20T14:30:02Z

bundle/apps/interpolate_variables_test.go

+	diags := bundle.Apply(context.Background(), b, InterpolateVariables())
+	require.Empty(t, diags)
+	require.Equal(t, []any([]any{map[string]any{"name": "JOB_ID", "value": "123"}}), b.Config.Resources.Apps["my_app_1"].Config["env"])
+	require.Nil(t, b.Config.Resources.Apps["my_app_2"].Config)


This test works but doesn't actually use the translation map.

I suspect you intend to make sure that the conversion of bundle-internal field references to Terraform resource field references is undone so that we can interpolate the IDs ourselves. But the test runs against a bundle-internal field reference.

Is there an alternative approach such that we don't have to convert back-and-forth? Maybe we completely skip this path during variable resolution? Or only skip resources.* references under resources.apps.*.config.**? Then we have less coupling with the TF resource interpolation.

pietern · 2024-12-20T14:32:57Z

bundle/apps/upload_config.go

+	var diags diag.Diagnostics
+	errGroup, ctx := errgroup.WithContext(ctx)
+
+	mu := &sync.Mutex{}


Doesn't need to be a pointer, can be a regular value.

pietern · 2024-12-20T14:34:00Z

bundle/apps/upload_config.go

+		// It means we need to write app.yml file with the content of the config field
+		// to the remote source code path of the app.
+		if app.Config != nil {
+			if !strings.HasPrefix(app.SourceCodePath, b.Config.Workspace.FilePath) {


Do we check this during validation as well?

pietern · 2024-12-20T14:34:21Z

bundle/apps/upload_config.go

+				return diag.FromErr(err)
+			}
+
+			// When the app is started, create a new app deployment and wait for it to complete.


Comment doesn't make sense here.

pietern · 2024-12-20T14:36:31Z

bundle/apps/validate.go

+			diags = append(diags, diag.Diagnostic{
+				Severity:  diag.Error,
+				Summary:   "Duplicate app source code path",
+				Detail:    fmt.Sprintf("app resource '%s' has the same source code path as app resource '%s'", key, usedSourceCodePaths[app.SourceCodePath]),


This should include why it is not allowed.

AFAICT it is because it causes a conflict on app.yml.

pietern · 2024-12-20T14:37:27Z

bundle/apps/validate.go

+		usedSourceCodePaths[app.SourceCodePath] = key
+
+		for _, configFile := range possibleConfigFiles {
+			cf := path.Join(app.SourceCodePath, configFile)


Is this guaranteed to be relative?

pietern · 2024-12-20T14:40:39Z

bundle/config/resources/apps.go

+}
+
+func (a *App) InitializeURL(baseURL url.URL) {
+	if a.Name == "" {


Discussed earlier; now that we don't have an ID, this will return an URL even if the app isn't deployed yet.

(confirmed this is the case with the latest build)

You could use the ModifiedStatus here.

pietern · 2024-12-20T14:44:57Z

bundle/deploy/terraform/util.go

@@ -34,6 +34,7 @@ type stateResourceInstance struct {

 type stateInstanceAttributes struct {
 	ID   string `json:"id"`
+	Name string `json:"name,omitempty"`


Can you call out that this is specifically for apps?

Other resources with a "name" will also see this field populated.

pietern · 2024-12-20T14:45:53Z

bundle/internal/schema/annotations.yml

+      PLACEHOLDER
+  "name":
+    "description": |-
+      PLACEHOLDER


Please run make schema with a path to the OpenAPI spec configured. I suspect these will show up in the OpenAPI annotations file instead and be non-empty. The apps API surface is documented.

cc @ilyakuz-db

That's correct, using genkit should also make the trick

I'll take a look how to make it more clear what to run if new fields were added. Looks like it should be possible to just filter annotations by package

pietern · 2024-12-20T14:54:18Z

integration/bundle/apps_test.go

+
+	// Try to run the app
+	_, out := runResourceWithStderr(t, ctx, root, "test_app")
+	require.Contains(t, out, app.Url)


How long does this take? Might be a candidate for testing.Short (ref: #2024).

andrewnester added the do-not-merge label Nov 21, 2024

andrewnester requested review from pietern, denik and shreyas-goenka November 21, 2024 16:26

andrewnester temporarily deployed to test-trigger-is November 21, 2024 16:26 — with GitHub Actions Inactive

andrewnester temporarily deployed to test-trigger-is November 21, 2024 16:33 — with GitHub Actions Inactive

andrewnester commented Nov 21, 2024

View reviewed changes

bundle/internal/tf/schema/resource_app.go Outdated Show resolved Hide resolved

andrewnester temporarily deployed to test-trigger-is November 21, 2024 16:36 — with GitHub Actions Inactive

andrewnester temporarily deployed to test-trigger-is November 21, 2024 16:46 — with GitHub Actions Inactive

andrewnester temporarily deployed to test-trigger-is November 26, 2024 15:14 — with GitHub Actions Inactive

andrewnester temporarily deployed to test-trigger-is November 26, 2024 15:15 — with GitHub Actions Inactive

andrewnester temporarily deployed to test-trigger-is November 26, 2024 15:17 — with GitHub Actions Inactive

andrewnester temporarily deployed to test-trigger-is November 27, 2024 14:11 — with GitHub Actions Inactive

andrewnester temporarily deployed to test-trigger-is November 28, 2024 14:33 — with GitHub Actions Inactive

andrewnester temporarily deployed to test-trigger-is November 28, 2024 15:39 — with GitHub Actions Inactive

andrewnester temporarily deployed to test-trigger-is November 28, 2024 15:40 — with GitHub Actions Inactive

andrewnester temporarily deployed to test-trigger-is November 28, 2024 15:41 — with GitHub Actions Inactive

andrewnester force-pushed the feature/apps branch from bc4c9ac to d72b03e Compare November 29, 2024 11:51

andrewnester temporarily deployed to test-trigger-is November 29, 2024 11:51 — with GitHub Actions Inactive

andrewnester temporarily deployed to test-trigger-is December 2, 2024 15:05 — with GitHub Actions Inactive

andrewnester added 13 commits December 18, 2024 12:28

addressed feedback

1a58f1b

interpolate after

2c359b4

use TF provider 1.61.0

a71289a

fixes after rebase

ddeeefe

fixes

1ff7bd2

test fixes

53393d5

fixed lint

e7a377c

fixed tests after rebase

6655eb6

fixed acc test

1ceec5e

removed test prefix

e8cae54

removed app id

ab85a7f

feedback

f6de4c1

added annotations

e9e0566

andrewnester force-pushed the feature/apps branch from 320b408 to e9e0566 Compare December 18, 2024 11:34

andrewnester temporarily deployed to test-trigger-is December 18, 2024 11:34 — with GitHub Actions Inactive

andrewnester temporarily deployed to test-trigger-is December 18, 2024 11:35 — with GitHub Actions Inactive

Added support for bundle generate and bind for Apps (#1946)

e622ab5

## Changes Added support for bundle generate and bind for Apps ## Tests - [ ] Add E2E test

andrewnester temporarily deployed to test-trigger-is December 18, 2024 12:16 — with GitHub Actions Inactive

fixed test for gcp

db516b6

andrewnester temporarily deployed to test-trigger-is December 18, 2024 13:34 — with GitHub Actions Inactive

denik reviewed Dec 19, 2024

View reviewed changes

bundle/apps/upload_config.go Show resolved Hide resolved

bundle/apps/upload_config.go Outdated Show resolved Hide resolved

bundle/apps/upload_config.go Show resolved Hide resolved

Added same source validation and made thread safe

2f8df81

andrewnester temporarily deployed to test-trigger-is December 19, 2024 13:00 — with GitHub Actions Inactive

andrewnester requested a review from denik December 20, 2024 11:58

pietern reviewed Dec 20, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added support for Databricks Apps in DABs #1928

Added support for Databricks Apps in DABs #1928

andrewnester commented Nov 21, 2024 •

edited

Loading

eng-dev-ecosystem-bot commented Dec 18, 2024

github-actions bot commented Dec 19, 2024

pietern Dec 20, 2024

pietern Dec 20, 2024

pietern Dec 20, 2024

pietern Dec 20, 2024

pietern Dec 20, 2024

pietern Dec 20, 2024

pietern Dec 20, 2024

pietern Dec 20, 2024

pietern Dec 20, 2024

pietern Dec 20, 2024

ilyakuz-db Dec 20, 2024

pietern Dec 20, 2024

Added support for Databricks Apps in DABs #1928

Are you sure you want to change the base?

Added support for Databricks Apps in DABs #1928

Conversation

andrewnester commented Nov 21, 2024 • edited Loading

Changes

Bundle configuration

Execution

Tests

eng-dev-ecosystem-bot commented Dec 18, 2024

github-actions bot commented Dec 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewnester commented Nov 21, 2024 •

edited

Loading