[Feature] Add `databricks_function` resource #4189

dgomez04 · 2024-11-02T02:35:33Z

Changes

Introduces databricks_function resource—allowing users to create and manage UDFs within Terraform. work in progress...

Closes #4074

Tests

make test run locally
relevant change in docs/ folder
covered with integration tests in internal/acceptance
relevant acceptance tests are passing
using Go SDK

…source updating feature branch.

ian-norris-ncino · 2024-11-04T16:47:15Z

Hello, is there any timeline on when this could make it into the provider? Would love to have this soon!

dgomez04 · 2024-11-04T19:36:56Z

Hi @ian-norris-ncino. I'm aiming to have this integrated into the provider by the end of the month, depending on my availability. I'll keep you posted.

alexott · 2024-11-16T11:43:11Z

docs/resources/function.md

+  name              = "calculate_bmi"
+  catalog_name      = databricks_catalog.sandbox.name
+  schema_name       = databricks_schema.functions.name
+  input_params      = [


It could be easier for people to specify input_param as separate blocks, like we do in other resources

Although doc says that input_params is a block, and then there are parameters inside: https://docs.databricks.com/api/workspace/functions/create#input_params

docs/resources/function.md

eng-dev-ecosystem-bot · 2024-11-17T12:46:34Z

Test Details: go/deco-tests/11869831592

internal/providers/pluginfw/resources/catalog/resource_function.go

tanmay-db

Hi @dgomez04, thanks for the PR, left some comments.

internal/providers/pluginfw/resources/catalog/resource_function.go

internal/providers/pluginfw/resources/catalog/resource_function_acc_test.go

…docs

github-actions · 2024-12-16T20:38:04Z

If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
go/deco-tests-run/terraform

Inputs:

PR number: 4189
Commit SHA: 7672d40d1b30f9fa98327fc547494919b2df3ed5

Checks will be approved automatically on success.

mgyucht

Looking nice! Some changes needed for the next iteration, but I think we're not far from the final state.

mgyucht · 2024-12-20T10:26:37Z

docs/resources/function.md

+resource "databricks_function" "calculate_bmi" {
+  name              = "calculate_bmi"
+  catalog_name      = databricks_catalog.sandbox.name
+  schema_name       = databricks_schema.functions.name
+  input_params      = [
+    {
+      name = "weight"
+      type = "DOUBLE"
+    },
+    {
+      name = "height"
+      type = "DOUBLE"
+    }
+  ]
+  data_type         = "DOUBLE"
+  routine_body      = "SQL"
+  routine_definition = "weight / (height * height)"
+  language          = "SQL"
+  is_deterministic  = true
+  sql_data_access   = "CONTAINS_SQL"
+  security_type     = "DEFINER"
+}


Based on my reading of the FunctionInfo type, this will need to look a bit different. Not your fault, as some of these things have changed since you started writing this PR. Specifically, the input_params is an object with a list nested object field parameters.

Suggested change

resource "databricks_function" "calculate_bmi" {

name = "calculate_bmi"

catalog_name = databricks_catalog.sandbox.name

schema_name = databricks_schema.functions.name

input_params = [

{

name = "weight"

type = "DOUBLE"

},

{

name = "height"

type = "DOUBLE"

}

]

data_type = "DOUBLE"

routine_body = "SQL"

routine_definition = "weight / (height * height)"

language = "SQL"

is_deterministic = true

sql_data_access = "CONTAINS_SQL"

security_type = "DEFINER"

}

resource "databricks_function" "calculate_bmi" {

name = "calculate_bmi"

catalog_name = databricks_catalog.sandbox.name

schema_name = databricks_schema.functions.name

input_params = {

parameters = [{

name = "weight"

type_name = "DOUBLE"

},

{

name = "height"

type_name = "DOUBLE"

}]

]

data_type = "DOUBLE"

routine_body = "SQL"

routine_definition = "weight / (height * height)"

is_deterministic = true

sql_data_access = "CONTAINS_SQL"

security_type = "DEFINER"

}

mgyucht · 2024-12-20T10:27:19Z

docs/resources/function.md

+resource "databricks_function" "calculate_bmi_py" {
+  name              = "calculate_bmi_py"
+  catalog_name      = databricks_catalog.sandbox.name
+  schema_name       = databricks_schema.functions.name
+  input_params      = [
+    {
+      name = "weight_kg"
+      type = "DOUBLE"
+    },
+    {
+      name = "height_m"
+      type = "DOUBLE"
+    }
+  ]
+  data_type         = "DOUBLE"
+  routine_body      = "EXTERNAL"
+  routine_definition = "return weight_kg / (height_m ** 2)"
+  language          = "Python"
+  is_deterministic  = false
+  sql_data_access   = "NO_SQL"
+  security_type     = "DEFINER"
+}


Suggested change

resource "databricks_function" "calculate_bmi_py" {

name = "calculate_bmi_py"

catalog_name = databricks_catalog.sandbox.name

schema_name = databricks_schema.functions.name

input_params = [

{

name = "weight_kg"

type = "DOUBLE"

},

{

name = "height_m"

type = "DOUBLE"

}

]

data_type = "DOUBLE"

routine_body = "EXTERNAL"

routine_definition = "return weight_kg / (height_m ** 2)"

language = "Python"

is_deterministic = false

sql_data_access = "NO_SQL"

security_type = "DEFINER"

}

resource "databricks_function" "calculate_bmi_py" {

name = "calculate_bmi_py"

catalog_name = databricks_catalog.sandbox.name

schema_name = databricks_schema.functions.name

input_params = {

parameters = [

{

name = "weight_kg"

type_name = "DOUBLE"

},

{

name = "height_m"

type_name = "DOUBLE"

}]

}

data_type = "DOUBLE"

routine_body = "EXTERNAL"

routine_definition = "return weight_kg / (height_m ** 2)"

is_deterministic = false

sql_data_access = "NO_SQL"

security_type = "DEFINER"

}

mgyucht · 2024-12-20T10:28:12Z

internal/providers/pluginfw/pluginfw_rollout_utils.go

+	// TODO Add resources here
+	sharing.ResourceShare, // Using the staging name (with pluginframework suffix)


This seems extra, can you remove this?

mgyucht · 2024-12-20T10:28:26Z

internal/providers/pluginfw/pluginfw_rollout_utils.go

@@ -42,6 +42,9 @@ var migratedDataSources = []func() datasource.DataSource{
 // List of resources that have been onboarded to the plugin framework - not migrated from sdkv2.
 // Keep this list sorted.
 var pluginFwOnlyResources = []func() resource.Resource{
+	// TODO Add resources here
+	sharing.ResourceShare, // Using the staging name (with pluginframework suffix)
+	catalog.ResourceFunction,


Can you sort this alphabetically (after app.ResourceApp)?

mgyucht · 2024-12-20T10:29:18Z

internal/providers/pluginfw/products/catalog/resource_function.go

+
+func (r *FunctionResource) Schema(ctx context.Context, req resource.SchemaRequest, resp *resource.SchemaResponse) {
+	attrs, blocks := tfschema.ResourceStructToSchemaMap(catalog_tf.FunctionInfo{}, func(c tfschema.CustomizableSchema) tfschema.CustomizableSchema {
+		c.ToNestedBlockObject()


You don't need to call this anymore (it was actually removed). For resources that are migrated from SDKv2, you need to call c.ConfigureForSdkV2Migration(), but this is a new resource, so nothing is needed here.

mgyucht · 2024-12-20T10:30:42Z

internal/providers/pluginfw/products/catalog/resource_function.go

+		c.SetRequired("name")
+		c.SetRequired("catalog_name")
+		c.SetRequired("schema_name")
+		c.SetRequired("input_params")
+		c.SetRequired("data_type")
+		c.SetRequired("full_data_type")
+		c.SetRequired("routine_defintion")
+		c.SetRequired("is_deterministic")
+		c.SetRequired("is_null_call")
+		c.SetRequired("specific_name")
+		c.SetRequired("routine_body")
+		c.SetRequired("security_type")
+		c.SetRequired("sql_data_access")
+		c.SetRequired("parameter_style")


Are these actually needed? As in, if you don't have them, are they treated as optional?

mgyucht · 2024-12-20T10:31:27Z

internal/providers/pluginfw/products/catalog/resource_function.go

+		return
+	}
+
+	funcName := stateFunc.Name.ValueString()


Do you need to use FullName here instead of Name?

mgyucht · 2024-12-20T10:34:45Z

internal/providers/pluginfw/products/catalog/resource_function.go

+		c.SetReadOnly("updated_at")
+		c.SetReadOnly("updated_by")
+
+		return c


I looked at UpdateFunction, and it seems like the only thing you can update about a function is its owner. Everything else cannot be updated. Can we mark all other fields with SetForceNew()?

dgomez04 added 5 commits October 30, 2024 18:15

started working on function resource.

27bd46d

started working on CRUD operations for functions resource.

f94dfd8

Added Read and Update methods for function resource

41d8d8f

Merge remote-tracking branch 'upstream/main' into feature/function-re…

20b2a4e

…source updating feature branch.

added all crud methods for function resource

2213438

dgomez04 requested review from a team as code owners November 2, 2024 02:35

dgomez04 requested review from hectorcast-db and removed request for a team November 2, 2024 02:35

dgomez04 temporarily deployed to test-trigger-is November 2, 2024 02:35 — with GitHub Actions Inactive

dgomez04 changed the title ~~[FEATURE] Add databricks_function resource~~ [FEATURE] WIP Add databricks_function resource Nov 2, 2024

dgomez04 changed the title ~~[FEATURE] WIP Add databricks_function resource~~ [Feature] [WIP] Add databricks_function resource Nov 2, 2024

dgomez04 added the do-not-merge label Nov 2, 2024

added acceptance tests for databricks_function resource

feba4e0

dgomez04 temporarily deployed to test-trigger-is November 7, 2024 00:47 — with GitHub Actions Inactive

added documentation for the databricks_function resource

2a46c05

dgomez04 temporarily deployed to test-trigger-is November 7, 2024 01:09 — with GitHub Actions Inactive

fixed typos in docs and added resource to the rollout utils

f0b2622

dgomez04 temporarily deployed to test-trigger-is November 7, 2024 01:12 — with GitHub Actions Inactive

dgomez04 removed the do-not-merge label Nov 7, 2024

dgomez04 changed the title ~~[Feature] [WIP] Add databricks_function resource~~ [Feature] Add databricks_function resource Nov 7, 2024

alexott requested review from rauchy and tanmay-db November 12, 2024 14:06

alexott requested changes Nov 16, 2024

View reviewed changes

alexott reviewed Nov 27, 2024

View reviewed changes

internal/providers/pluginfw/resources/catalog/resource_function.go Outdated Show resolved Hide resolved

internal/providers/pluginfw/resources/catalog/resource_function.go Outdated Show resolved Hide resolved

tanmay-db reviewed Nov 27, 2024

View reviewed changes

dgomez04 added 5 commits December 2, 2024 11:38

merged with main branch

ca2a54c

added requested changes and included an update pathway in the tests

64b136c

added missing required attributes and finished arugment reference on …

d23cdc4

…docs

removed waiting time for function resource

fcd33dd

fixed merge conflicts

7672d40

mgyucht requested changes Dec 20, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Add `databricks_function` resource #4189

[Feature] Add `databricks_function` resource #4189

dgomez04 commented Nov 2, 2024 •

edited by alexott

Loading

ian-norris-ncino commented Nov 4, 2024

dgomez04 commented Nov 4, 2024

alexott Nov 16, 2024

alexott Nov 16, 2024

eng-dev-ecosystem-bot commented Nov 17, 2024

tanmay-db left a comment

github-actions bot commented Dec 16, 2024

mgyucht left a comment

mgyucht Dec 20, 2024

mgyucht Dec 20, 2024

mgyucht Dec 20, 2024

mgyucht Dec 20, 2024

mgyucht Dec 20, 2024

mgyucht Dec 20, 2024

mgyucht Dec 20, 2024

mgyucht Dec 20, 2024

		// TODO Add resources here
		sharing.ResourceShare, // Using the staging name (with pluginframework suffix)

[Feature] Add databricks_function resource #4189

Are you sure you want to change the base?

[Feature] Add databricks_function resource #4189

Conversation

dgomez04 commented Nov 2, 2024 • edited by alexott Loading

Changes

Tests

ian-norris-ncino commented Nov 4, 2024

dgomez04 commented Nov 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eng-dev-ecosystem-bot commented Nov 17, 2024

tanmay-db left a comment

Choose a reason for hiding this comment

github-actions bot commented Dec 16, 2024

mgyucht left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

[Feature] Add `databricks_function` resource #4189

[Feature] Add `databricks_function` resource #4189

dgomez04 commented Nov 2, 2024 •

edited by alexott

Loading