refactor: resource bounded context #650
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This reverts commit e863905.
Co-authored-by: Sandeep Bhardwaj <[email protected]>
sbchaos
previously approved these changes
Nov 29, 2022
irainia
commented
Nov 30, 2022
core/resource/resource.go
Outdated
| @@ -233,7 +233,7 @@ func (r *Resource) MarkSkipped() error { | |||
| } | |||
|
|
|||
| func (r *Resource) MarkToCreate() error { | |||
| if r.status == StatusValidationSuccess || r.status == StatusCreateFailure { | |||
| if r.status == StatusValidationSuccess { | |||
There was a problem hiding this comment.
What is the reason of changing the requirement to transition status into to_create only status success? I think it's a bit different than the last discussion.
There was a problem hiding this comment.
There are issues in previous version. Let's discuss this.
core/resource/resource.go
Outdated
| @@ -243,7 +243,7 @@ func (r *Resource) MarkToCreate() error { | |||
|
|
|||
| func (r *Resource) MarkToUpdate() error { | |||
| switch r.status { | |||
| case StatusValidationSuccess, StatusUpdateFailure, StatusSuccess, StatusExistInStore: | |||
| case StatusValidationSuccess: | |||
There was a problem hiding this comment.
Also for to_update, perhaps we can discuss about this as it's a bit different than the last discussion.
There was a problem hiding this comment.
Sure, let's have a discussion.
| _ = incoming.MarkToUpdate() | ||
| } | ||
|
|
||
| err := rs.repo.Update(ctx, incoming) |
There was a problem hiding this comment.
Maybe we can move this Update repo call into outside of this function getResourcesToBatchUpdate; or we change the name getResourcesToBatchUpdate into something else like updateResourcesForDeploy. Because if we don't do any changes, then the function will not reflect what it actually does.
sbchaos
added a commit
that referenced
this pull request
Dec 5, 2022
sbchaos
added a commit
that referenced
this pull request
Dec 5, 2022
* wip: add domain models for resource bounded context * refactor: add handle and service for resources * refactor: add validate for resources * refactor: add converts for resource * refactor: update resource api * refactor: add test for resource handler * test: add test for metadata and dataset * test: add test for resource models * test: add test for resource * refactor: add resource test and fix lint issues * feat: add resource service * refactor: rename methods in resource repository for better clarity * feat: add batch update on resource service * refactor: simplify interface being used by service and update status * refactor: aedd extra config to resources * refactor: add resource manager * refactor: add bigquery handler for dataset, view and ext_table * refactor: define all methods on pointer receiver * feat: add resource replacement migration * feat: add function to convert string to status * feat: add resource repository * refactor: add table apis * refactor: add bigquery store * fix: fix tests for resources * refactor: unit test for resource repository to be independent from each other * test: refactor resource unit test to cover more cases * test: refactor service unit test for better readability * refactor: add exists check on resource handle * refactor: add batch update and multi error * feat: add index on resource using project namd and namespace name * feat: add tenant checks before creating resource * refactor: finish the code for batch update * fix: old resource usage error in backup repository * fix: fix failing test * test: update repository test to use oen parallelism * refactor: rename bigquery store from BigQuery -> Bigquery Co-authored-by: Anwar Hidayat <[email protected]> * refactor: resource handle accepts only the resource Co-authored-by: Anwar Hidayat <[email protected]> * refactor: wire the dependencies for resource handler Co-authored-by: Anwar Hidayat <[email protected]> * feat: add multi-error converter Co-authored-by: Sandeep Bhardwaj <[email protected]> * fix: add multi-to error converter Co-authored-by: Sandeep Bhardwaj <[email protected]> * refactor: update repository to allow for partial update and add unit test Co-authored-by: Sandeep Bhardwaj <[email protected]> * refactor: update the check for status * test: add test for resource manager * test: add test for resource models * fix: bug cannot find an existing secret Co-authored-by: Sandeep Bhardwaj <[email protected]> * fix: batch update behavior to create or update based on the provided status * feat: add telemetry for key resource implementations * fix: fix issues with batch update * test: add test for batch * fix: fix lint errors * refactor: remove unused function * refactor: add logger wrapper for server to simplify log * feat: add process logging in resource service * feat: add process logging in resource manager * fix: add log writer for resource service and handler * feat: add backup sub context * chore: change log level from warning to error for resource deploy in handler * refactor: Revert "feat: add telemetry for key resource implementations" This reverts commit e416fa6. * refactor: add some status mutation methods for resource * test: update unit test for mark status to use resource creation from existing Retaining the usage of function 'FromExisting' is necessary. One might argue that this is not a really good approach. Yes, in a certain cases it is not a good approach. An example in which we no longer use this function is when we are doing resource deployment, which requires update status. However, there are certain cases where we still need this function as a way for shortcut in setting status. An obvious example is when we are fetching resource in the database. The way we store it is in a string format, thus we need to convert it. Nonetheless, we don't know what status it is. And during the resource initialiation, it is initialized with 'unknown' status. In order to 'reach' the status from database, we have to perform some actions on marking statuses so it will become the same. This is tricky, and even the tricky part is we don't know how many steps we have to perform. As a side note, we should limit its usage as much as possible. * fix: remove resource status from equality factor * refactor: remove log writer usage in service layer * fix: allow reading error from repository when batch updating * refactor: restructure how to send response through log writer by prioritizing the success, skipped, error, and the errors * refactor: add logger other than writer when writing resource status * feat: add metric for batch update on skipped, success, and failure resources * fix: resource equality bug on tenant * fix: bug on empty resources for store batch update * fix: database migration conflict * fix: bug on skipped resource regardless of the status * feat: use complete message when encountering error during resource deployment * lint: fix linter issue * fix: re-order status to address additional cases * fix: make sure the resources to be processed have success validation status * fix: syntax and unit test error Co-authored-by: Sandeep Bhardwaj <[email protected]> * refactor: add models for backup result * refactor: redesign resource status note: this commit still contains both syntactical and logical error * refactor: rename batch update to deploy based on discussion note: this commit still contains both syntactical and logical error * refactor: update resource manage to check whether resource exist in store or not note: this commit contains both syntactical and logical error * feat: add store existence status in resource * refactor: add update status on repo for resource in service note: this commit contains both syntactical and logical error * chore: update the proton commit * refactor: update proton commit * refactor: add handler methods and test * refactor: add service for backup * fix: add create failure and update failure status precedence for transition * refactor: re-implement service for create resource * refactor: update how status is changed through the transition for better readbility and maintenance * refactor: modify create resource to return multi error * fix: allows status transition to the current status * refactor: re-implement service for update resource * fix: add resource status in store * refactor: remove state transition management to simplify this removal is because the state transition changes can move to any status. concrete example of this is status 'unknown'. this status can change to any status based on the current service requirements. so, for now it will be removed. * chore: update protos * refactor: use backup model * fix: status bug on create and update resource * refactor: re-implement service for deploy resources * fix: status transition for repository * feat: add existence check on resource manager * feat: add existence check on bigquery datastore * fix: syntax error caused by missed contract * fix: linter issue in service * refactor: add migrations and repository for backup * refactor: add repository methods for resources * fix: fix the client to use new protos * refactor: wire up new implementation and remove unused code * refactor: simplify reading store name * refactor: change way to create resource handels * refactor: add extra methods to table resource handle * refactor: add backup for tables * Revert "fix: linter issue in service" This reverts commit 56de127. * Revert "fix: syntax error caused by missed contract" This reverts commit f970be1. * Revert "feat: add existence check on bigquery datastore" This reverts commit 12ed802. * Revert "feat: add existence check on resource manager" This reverts commit 0cb92aa. * Revert "fix: status transition for repository" This reverts commit 4c0fa25. * Revert "refactor: re-implement service for deploy resources" This reverts commit 084f316. * Revert "fix: status bug on create and update resource" This reverts commit cc3111a. * Revert "refactor: remove state transition management to simplify" This reverts commit 697488b. * Revert "fix: add resource status in store" This reverts commit 75e2a45. * Revert "refactor: re-implement service for update resource" This reverts commit 9c163df. * Revert "fix: allows status transition to the current status" This reverts commit 1fb76e6. * Revert "refactor: modify create resource to return multi error" This reverts commit 6ad9634. * Revert "refactor: update how status is changed through the transition for better readbility and maintenance" This reverts commit 90aa2ed. * Revert "refactor: re-implement service for create resource" This reverts commit f4bc384. * Revert "fix: add create failure and update failure status precedence for transition" This reverts commit 77eb5bb. * Revert "refactor: add update status on repo for resource in service" This reverts commit 6970b32. * Revert "feat: add store existence status in resource" This reverts commit cd9520d. * Revert "refactor: update resource manage to check whether resource exist in store or not" This reverts commit 0f0ea9c. * Revert "refactor: rename batch update to deploy based on discussion" This reverts commit bfe9029. * Revert "refactor: redesign resource status" This reverts commit e863905. * refactor: update available statuses based on the latest convention * fix: update status transition based on the latest convention * refactor: rename batch update to deploy based on the latest discussion * refactor: re-implement resource create service based on the latest state transition * test: add test for backup * fix: lint errors * fix: resource manager to adopt the latest status transition Co-authored-by: Sandeep Bhardwaj <[email protected]> * fix: catch possible intermittent error during getting existing resource * fix: use custom not found error if encountered db not found error * fix: update resource service based on the latest state transition * fix: use float64 for proto values and not int * test: add missing tests * refactor: check resource status on deploy * test: add test * fix: fix lint issues * fix: fix flaky test * fix: fix issues while qa * fix: trim the resource name before sending * rework: rework the location of resources * fix: fix error message * fix: fix failing test * refactor: get urn from datastore * refactor: update urn in store * refactor: use resources kind as string * refactor: delete bigquery adapter * refactor: remove references to datastore * refactor: resource bounded context (#650) * wip: add domain models for resource bounded context * refactor: add handle and service for resources * refactor: add validate for resources * refactor: add converts for resource * refactor: update resource api * refactor: add test for resource handler * test: add test for metadata and dataset * test: add test for resource models * test: add test for resource * refactor: add resource test and fix lint issues * feat: add resource service * refactor: rename methods in resource repository for better clarity * feat: add batch update on resource service * refactor: simplify interface being used by service and update status * refactor: aedd extra config to resources * refactor: add resource manager * refactor: add bigquery handler for dataset, view and ext_table * refactor: define all methods on pointer receiver * feat: add resource replacement migration * feat: add function to convert string to status * feat: add resource repository * refactor: add table apis * refactor: add bigquery store * fix: fix tests for resources * refactor: unit test for resource repository to be independent from each other * test: refactor resource unit test to cover more cases * test: refactor service unit test for better readability * refactor: add exists check on resource handle * refactor: add batch update and multi error * feat: add index on resource using project namd and namespace name * feat: add tenant checks before creating resource * refactor: finish the code for batch update * fix: old resource usage error in backup repository * fix: fix failing test * test: update repository test to use oen parallelism * refactor: rename bigquery store from BigQuery -> Bigquery Co-authored-by: Anwar Hidayat <[email protected]> * refactor: resource handle accepts only the resource Co-authored-by: Anwar Hidayat <[email protected]> * refactor: wire the dependencies for resource handler Co-authored-by: Anwar Hidayat <[email protected]> * feat: add multi-error converter Co-authored-by: Sandeep Bhardwaj <[email protected]> * fix: add multi-to error converter Co-authored-by: Sandeep Bhardwaj <[email protected]> * refactor: update repository to allow for partial update and add unit test Co-authored-by: Sandeep Bhardwaj <[email protected]> * refactor: update the check for status * test: add test for resource manager * test: add test for resource models * fix: bug cannot find an existing secret Co-authored-by: Sandeep Bhardwaj <[email protected]> * fix: batch update behavior to create or update based on the provided status * feat: add telemetry for key resource implementations * fix: fix issues with batch update * test: add test for batch * fix: fix lint errors * refactor: remove unused function * refactor: add logger wrapper for server to simplify log * feat: add process logging in resource service * feat: add process logging in resource manager * fix: add log writer for resource service and handler * chore: change log level from warning to error for resource deploy in handler * refactor: Revert "feat: add telemetry for key resource implementations" This reverts commit e416fa6. * refactor: add some status mutation methods for resource * test: update unit test for mark status to use resource creation from existing Retaining the usage of function 'FromExisting' is necessary. One might argue that this is not a really good approach. Yes, in a certain cases it is not a good approach. An example in which we no longer use this function is when we are doing resource deployment, which requires update status. However, there are certain cases where we still need this function as a way for shortcut in setting status. An obvious example is when we are fetching resource in the database. The way we store it is in a string format, thus we need to convert it. Nonetheless, we don't know what status it is. And during the resource initialiation, it is initialized with 'unknown' status. In order to 'reach' the status from database, we have to perform some actions on marking statuses so it will become the same. This is tricky, and even the tricky part is we don't know how many steps we have to perform. As a side note, we should limit its usage as much as possible. * fix: remove resource status from equality factor * refactor: remove log writer usage in service layer * fix: allow reading error from repository when batch updating * refactor: restructure how to send response through log writer by prioritizing the success, skipped, error, and the errors * refactor: add logger other than writer when writing resource status * feat: add metric for batch update on skipped, success, and failure resources * fix: resource equality bug on tenant * fix: bug on empty resources for store batch update * fix: database migration conflict * fix: bug on skipped resource regardless of the status * feat: use complete message when encountering error during resource deployment * lint: fix linter issue * fix: re-order status to address additional cases * fix: make sure the resources to be processed have success validation status * fix: syntax and unit test error Co-authored-by: Sandeep Bhardwaj <[email protected]> * refactor: redesign resource status note: this commit still contains both syntactical and logical error * refactor: rename batch update to deploy based on discussion note: this commit still contains both syntactical and logical error * refactor: update resource manage to check whether resource exist in store or not note: this commit contains both syntactical and logical error * feat: add store existence status in resource * refactor: add update status on repo for resource in service note: this commit contains both syntactical and logical error * fix: add create failure and update failure status precedence for transition * refactor: re-implement service for create resource * refactor: update how status is changed through the transition for better readbility and maintenance * refactor: modify create resource to return multi error * fix: allows status transition to the current status * refactor: re-implement service for update resource * fix: add resource status in store * refactor: remove state transition management to simplify this removal is because the state transition changes can move to any status. concrete example of this is status 'unknown'. this status can change to any status based on the current service requirements. so, for now it will be removed. * fix: status bug on create and update resource * refactor: re-implement service for deploy resources * fix: status transition for repository * feat: add existence check on resource manager * feat: add existence check on bigquery datastore * fix: syntax error caused by missed contract * fix: linter issue in service * Revert "fix: linter issue in service" This reverts commit 56de127. * Revert "fix: syntax error caused by missed contract" This reverts commit f970be1. * Revert "feat: add existence check on bigquery datastore" This reverts commit 12ed802. * Revert "feat: add existence check on resource manager" This reverts commit 0cb92aa. * Revert "fix: status transition for repository" This reverts commit 4c0fa25. * Revert "refactor: re-implement service for deploy resources" This reverts commit 084f316. * Revert "fix: status bug on create and update resource" This reverts commit cc3111a. * Revert "refactor: remove state transition management to simplify" This reverts commit 697488b. * Revert "fix: add resource status in store" This reverts commit 75e2a45. * Revert "refactor: re-implement service for update resource" This reverts commit 9c163df. * Revert "fix: allows status transition to the current status" This reverts commit 1fb76e6. * Revert "refactor: modify create resource to return multi error" This reverts commit 6ad9634. * Revert "refactor: update how status is changed through the transition for better readbility and maintenance" This reverts commit 90aa2ed. * Revert "refactor: re-implement service for create resource" This reverts commit f4bc384. * Revert "fix: add create failure and update failure status precedence for transition" This reverts commit 77eb5bb. * Revert "refactor: add update status on repo for resource in service" This reverts commit 6970b32. * Revert "feat: add store existence status in resource" This reverts commit cd9520d. * Revert "refactor: update resource manage to check whether resource exist in store or not" This reverts commit 0f0ea9c. * Revert "refactor: rename batch update to deploy based on discussion" This reverts commit bfe9029. * Revert "refactor: redesign resource status" This reverts commit e863905. * refactor: update available statuses based on the latest convention * fix: update status transition based on the latest convention * refactor: rename batch update to deploy based on the latest discussion * refactor: re-implement resource create service based on the latest state transition * fix: resource manager to adopt the latest status transition Co-authored-by: Sandeep Bhardwaj <[email protected]> * fix: catch possible intermittent error during getting existing resource * fix: use custom not found error if encountered db not found error * fix: update resource service based on the latest state transition * fix: use float64 for proto values and not int * refactor: check resource status on deploy * test: add test * fix: fix lint issues * fix: fix flaky test * rework: rework the location of resources * refactor: get urn from datastore * refactor: update urn in store * refactor: use resources kind as string Co-authored-by: Sandeep Bhardwaj <[email protected]> * chore: update proton commit Co-authored-by: Anwar Hidayat <[email protected]> Co-authored-by: Anwar Hidayat <[email protected]>
sbchaos
added a commit
that referenced
this pull request
Jan 2, 2023
* refactor: resource bounded context (#675) * wip: add domain models for resource bounded context * refactor: add handle and service for resources * refactor: add validate for resources * refactor: add converts for resource * refactor: update resource api * refactor: add test for resource handler * test: add test for metadata and dataset * test: add test for resource models * test: add test for resource * refactor: add resource test and fix lint issues * feat: add resource service * refactor: rename methods in resource repository for better clarity * feat: add batch update on resource service * refactor: simplify interface being used by service and update status * refactor: aedd extra config to resources * refactor: add resource manager * refactor: add bigquery handler for dataset, view and ext_table * refactor: define all methods on pointer receiver * feat: add resource replacement migration * feat: add function to convert string to status * feat: add resource repository * refactor: add table apis * refactor: add bigquery store * fix: fix tests for resources * refactor: unit test for resource repository to be independent from each other * test: refactor resource unit test to cover more cases * test: refactor service unit test for better readability * refactor: add exists check on resource handle * refactor: add batch update and multi error * feat: add index on resource using project namd and namespace name * feat: add tenant checks before creating resource * refactor: finish the code for batch update * fix: old resource usage error in backup repository * fix: fix failing test * test: update repository test to use oen parallelism * refactor: rename bigquery store from BigQuery -> Bigquery Co-authored-by: Anwar Hidayat <[email protected]> * refactor: resource handle accepts only the resource Co-authored-by: Anwar Hidayat <[email protected]> * refactor: wire the dependencies for resource handler Co-authored-by: Anwar Hidayat <[email protected]> * feat: add multi-error converter Co-authored-by: Sandeep Bhardwaj <[email protected]> * fix: add multi-to error converter Co-authored-by: Sandeep Bhardwaj <[email protected]> * refactor: update repository to allow for partial update and add unit test Co-authored-by: Sandeep Bhardwaj <[email protected]> * refactor: update the check for status * test: add test for resource manager * test: add test for resource models * fix: bug cannot find an existing secret Co-authored-by: Sandeep Bhardwaj <[email protected]> * fix: batch update behavior to create or update based on the provided status * feat: add telemetry for key resource implementations * fix: fix issues with batch update * test: add test for batch * fix: fix lint errors * refactor: remove unused function * refactor: add logger wrapper for server to simplify log * feat: add process logging in resource service * feat: add process logging in resource manager * fix: add log writer for resource service and handler * chore: change log level from warning to error for resource deploy in handler * refactor: Revert "feat: add telemetry for key resource implementations" This reverts commit e416fa6a5d63aa1d57e8ee78c1659658dd5e6cd5. * refactor: add some status mutation methods for resource * test: update unit test for mark status to use resource creation from existing Retaining the usage of function 'FromExisting' is necessary. One might argue that this is not a really good approach. Yes, in a certain cases it is not a good approach. An example in which we no longer use this function is when we are doing resource deployment, which requires update status. However, there are certain cases where we still need this function as a way for shortcut in setting status. An obvious example is when we are fetching resource in the database. The way we store it is in a string format, thus we need to convert it. Nonetheless, we don't know what status it is. And during the resource initialiation, it is initialized with 'unknown' status. In order to 'reach' the status from database, we have to perform some actions on marking statuses so it will become the same. This is tricky, and even the tricky part is we don't know how many steps we have to perform. As a side note, we should limit its usage as much as possible. * fix: remove resource status from equality factor * refactor: remove log writer usage in service layer * fix: allow reading error from repository when batch updating * refactor: restructure how to send response through log writer by prioritizing the success, skipped, error, and the errors * refactor: add logger other than writer when writing resource status * feat: add metric for batch update on skipped, success, and failure resources * fix: resource equality bug on tenant * fix: bug on empty resources for store batch update * fix: database migration conflict * fix: bug on skipped resource regardless of the status * feat: use complete message when encountering error during resource deployment * lint: fix linter issue * fix: re-order status to address additional cases * fix: make sure the resources to be processed have success validation status * fix: syntax and unit test error Co-authored-by: Sandeep Bhardwaj <[email protected]> * refactor: redesign resource status note: this commit still contains both syntactical and logical error * refactor: rename batch update to deploy based on discussion note: this commit still contains both syntactical and logical error * refactor: update resource manage to check whether resource exist in store or not note: this commit contains both syntactical and logical error * feat: add store existence status in resource * refactor: add update status on repo for resource in service note: this commit contains both syntactical and logical error * fix: add create failure and update failure status precedence for transition * refactor: re-implement service for create resource * refactor: update how status is changed through the transition for better readbility and maintenance * refactor: modify create resource to return multi error * fix: allows status transition to the current status * refactor: re-implement service for update resource * fix: add resource status in store * refactor: remove state transition management to simplify this removal is because the state transition changes can move to any status. concrete example of this is status 'unknown'. this status can change to any status based on the current service requirements. so, for now it will be removed. * fix: status bug on create and update resource * refactor: re-implement service for deploy resources * fix: status transition for repository * feat: add existence check on resource manager * feat: add existence check on bigquery datastore * fix: syntax error caused by missed contract * fix: linter issue in service * Revert "fix: linter issue in service" This reverts commit 56de127a5c47c0fdf5b231d5fdde3955340c939e. * Revert "fix: syntax error caused by missed contract" This reverts commit f970be1d84acd28d1a273c348468752d3fcc2da0. * Revert "feat: add existence check on bigquery datastore" This reverts commit 12ed8020854a9842a890c9af1e8e49b7384c15d8. * Revert "feat: add existence check on resource manager" This reverts commit 0cb92aa308d203d1bbd375e775d417e826de6448. * Revert "fix: status transition for repository" This reverts commit 4c0fa25f2deb1ff735022bb9fc86c571ee92bc4d. * Revert "refactor: re-implement service for deploy resources" This reverts commit 084f316b3b731389e2a1291383c02094d90d6905. * Revert "fix: status bug on create and update resource" This reverts commit cc3111af88549b8e2b3eed61e7fc429bbc7cf7cb. * Revert "refactor: remove state transition management to simplify" This reverts commit 697488bd5a2639237f8909b260a374e77b8520a9. * Revert "fix: add resource status in store" This reverts commit 75e2a45bc60fcb9139780c7ce11cff40e502076c. * Revert "refactor: re-implement service for update resource" This reverts commit 9c163df4477823d0f02d726c7989d02e5cc7ca74. * Revert "fix: allows status transition to the current status" This reverts commit 1fb76e610258c6d0ed0497af8a6b9c4514ca77ed. * Revert "refactor: modify create resource to return multi error" This reverts commit 6ad9634d525c29126bc9693150fc3e9c9091b7e4. * Revert "refactor: update how status is changed through the transition for better readbility and maintenance" This reverts commit 90aa2ede335c4f068d1ac0599f072b0107f94533. * Revert "refactor: re-implement service for create resource" This reverts commit f4bc3840441b7364f533c952a5d656a6698ccaee. * Revert "fix: add create failure and update failure status precedence for transition" This reverts commit 77eb5bba6758c109f89ff54441bd38e693012140. * Revert "refactor: add update status on repo for resource in service" This reverts commit 6970b32878af7cc02a538dac127b53de9b7fd573. * Revert "feat: add store existence status in resource" This reverts commit cd9520df8e0cf7feed2b0817cf8be1fa15929138. * Revert "refactor: update resource manage to check whether resource exist in store or not" This reverts commit 0f0ea9cbcc63e19d196e110ea2c34ef488586c99. * Revert "refactor: rename batch update to deploy based on discussion" This reverts commit bfe90293956416209b91e4f6ce8c2532c0604bfe. * Revert "refactor: redesign resource status" This reverts commit e863905d7aed16a68c0601965ddb4dd1022dabce. * refactor: update available statuses based on the latest convention * fix: update status transition based on the latest convention * refactor: rename batch update to deploy based on the latest discussion * refactor: re-implement resource create service based on the latest state transition * fix: resource manager to adopt the latest status transition Co-authored-by: Sandeep Bhardwaj <[email protected]> * fix: catch possible intermittent error during getting existing resource * fix: use custom not found error if encountered db not found error * fix: update resource service based on the latest state transition * fix: use float64 for proto values and not int * refactor: check resource status on deploy * test: add test * fix: fix lint issues * fix: fix flaky test * rework: rework the location of resources * refactor: get urn from datastore * refactor: update urn in store * refactor: use resources kind as string Co-authored-by: Anwar Hidayat <[email protected]> * refactor: backup sub context for resources (#664) * wip: add domain models for resource bounded context * refactor: add handle and service for resources * refactor: add validate for resources * refactor: add converts for resource * refactor: update resource api * refactor: add test for resource handler * test: add test for metadata and dataset * test: add test for resource models * test: add test for resource * refactor: add resource test and fix lint issues * feat: add resource service * refactor: rename methods in resource repository for better clarity * feat: add batch update on resource service * refactor: simplify interface being used by service and update status * refactor: aedd extra config to resources * refactor: add resource manager * refactor: add bigquery handler for dataset, view and ext_table * refactor: define all methods on pointer receiver * feat: add resource replacement migration * feat: add function to convert string to status * feat: add resource repository * refactor: add table apis * refactor: add bigquery store * fix: fix tests for resources * refactor: unit test for resource repository to be independent from each other * test: refactor resource unit test to cover more cases * test: refactor service unit test for better readability * refactor: add exists check on resource handle * refactor: add batch update and multi error * feat: add index on resource using project namd and namespace name * feat: add tenant checks before creating resource * refactor: finish the code for batch update * fix: old resource usage error in backup repository * fix: fix failing test * test: update repository test to use oen parallelism * refactor: rename bigquery store from BigQuery -> Bigquery Co-authored-by: Anwar Hidayat <[email protected]> * refactor: resource handle accepts only the resource Co-authored-by: Anwar Hidayat <[email protected]> * refactor: wire the dependencies for resource handler Co-authored-by: Anwar Hidayat <[email protected]> * feat: add multi-error converter Co-authored-by: Sandeep Bhardwaj <[email protected]> * fix: add multi-to error converter Co-authored-by: Sandeep Bhardwaj <[email protected]> * refactor: update repository to allow for partial update and add unit test Co-authored-by: Sandeep Bhardwaj <[email protected]> * refactor: update the check for status * test: add test for resource manager * test: add test for resource models * fix: bug cannot find an existing secret Co-authored-by: Sandeep Bhardwaj <[email protected]> * fix: batch update behavior to create or update based on the provided status * feat: add telemetry for key resource implementations * fix: fix issues with batch update * test: add test for batch * fix: fix lint errors * refactor: remove unused function * refactor: add logger wrapper for server to simplify log * feat: add process logging in resource service * feat: add process logging in resource manager * fix: add log writer for resource service and handler * feat: add backup sub context * chore: change log level from warning to error for resource deploy in handler * refactor: Revert "feat: add telemetry for key resource implementations" This reverts commit e416fa6a5d63aa1d57e8ee78c1659658dd5e6cd5. * refactor: add some status mutation methods for resource * test: update unit test for mark status to use resource creation from existing Retaining the usage of function 'FromExisting' is necessary. One might argue that this is not a really good approach. Yes, in a certain cases it is not a good approach. An example in which we no longer use this function is when we are doing resource deployment, which requires update status. However, there are certain cases where we still need this function as a way for shortcut in setting status. An obvious example is when we are fetching resource in the database. The way we store it is in a string format, thus we need to convert it. Nonetheless, we don't know what status it is. And during the resource initialiation, it is initialized with 'unknown' status. In order to 'reach' the status from database, we have to perform some actions on marking statuses so it will become the same. This is tricky, and even the tricky part is we don't know how many steps we have to perform. As a side note, we should limit its usage as much as possible. * fix: remove resource status from equality factor * refactor: remove log writer usage in service layer * fix: allow reading error from repository when batch updating * refactor: restructure how to send response through log writer by prioritizing the success, skipped, error, and the errors * refactor: add logger other than writer when writing resource status * feat: add metric for batch update on skipped, success, and failure resources * fix: resource equality bug on tenant * fix: bug on empty resources for store batch update * fix: database migration conflict * fix: bug on skipped resource regardless of the status * feat: use complete message when encountering error during resource deployment * lint: fix linter issue * fix: re-order status to address additional cases * fix: make sure the resources to be processed have success validation status * fix: syntax and unit test error Co-authored-by: Sandeep Bhardwaj <[email protected]> * refactor: add models for backup result * refactor: redesign resource status note: this commit still contains both syntactical and logical error * refactor: rename batch update to deploy based on discussion note: this commit still contains both syntactical and logical error * refactor: update resource manage to check whether resource exist in store or not note: this commit contains both syntactical and logical error * feat: add store existence status in resource * refactor: add update status on repo for resource in service note: this commit contains both syntactical and logical error * chore: update the proton commit * refactor: update proton commit * refactor: add handler methods and test * refactor: add service for backup * fix: add create failure and update failure status precedence for transition * refactor: re-implement service for create resource * refactor: update how status is changed through the transition for better readbility and maintenance * refactor: modify create resource to return multi error * fix: allows status transition to the current status * refactor: re-implement service for update resource * fix: add resource status in store * refactor: remove state transition management to simplify this removal is because the state transition changes can move to any status. concrete example of this is status 'unknown'. this status can change to any status based on the current service requirements. so, for now it will be removed. * chore: update protos * refactor: use backup model * fix: status bug on create and update resource * refactor: re-implement service for deploy resources * fix: status transition for repository * feat: add existence check on resource manager * feat: add existence check on bigquery datastore * fix: syntax error caused by missed contract * fix: linter issue in service * refactor: add migrations and repository for backup * refactor: add repository methods for resources * fix: fix the client to use new protos * refactor: wire up new implementation and remove unused code * refactor: simplify reading store name * refactor: change way to create resource handels * refactor: add extra methods to table resource handle * refactor: add backup for tables * Revert "fix: linter issue in service" This reverts commit 56de127a5c47c0fdf5b231d5fdde3955340c939e. * Revert "fix: syntax error caused by missed contract" This reverts commit f970be1d84acd28d1a273c348468752d3fcc2da0. * Revert "feat: add existence check on bigquery datastore" This reverts commit 12ed8020854a9842a890c9af1e8e49b7384c15d8. * Revert "feat: add existence check on resource manager" This reverts commit 0cb92aa308d203d1bbd375e775d417e826de6448. * Revert "fix: status transition for repository" This reverts commit 4c0fa25f2deb1ff735022bb9fc86c571ee92bc4d. * Revert "refactor: re-implement service for deploy resources" This reverts commit 084f316b3b731389e2a1291383c02094d90d6905. * Revert "fix: status bug on create and update resource" This reverts commit cc3111af88549b8e2b3eed61e7fc429bbc7cf7cb. * Revert "refactor: remove state transition management to simplify" This reverts commit 697488bd5a2639237f8909b260a374e77b8520a9. * Revert "fix: add resource status in store" This reverts commit 75e2a45bc60fcb9139780c7ce11cff40e502076c. * Revert "refactor: re-implement service for update resource" This reverts commit 9c163df4477823d0f02d726c7989d02e5cc7ca74. * Revert "fix: allows status transition to the current status" This reverts commit 1fb76e610258c6d0ed0497af8a6b9c4514ca77ed. * Revert "refactor: modify create resource to return multi error" This reverts commit 6ad9634d525c29126bc9693150fc3e9c9091b7e4. * Revert "refactor: update how status is changed through the transition for better readbility and maintenance" This reverts commit 90aa2ede335c4f068d1ac0599f072b0107f94533. * Revert "refactor: re-implement service for create resource" This reverts commit f4bc3840441b7364f533c952a5d656a6698ccaee. * Revert "fix: add create failure and update failure status precedence for transition" This reverts commit 77eb5bba6758c109f89ff54441bd38e693012140. * Revert "refactor: add update status on repo for resource in service" This reverts commit 6970b32878af7cc02a538dac127b53de9b7fd573. * Revert "feat: add store existence status in resource" This reverts commit cd9520df8e0cf7feed2b0817cf8be1fa15929138. * Revert "refactor: update resource manage to check whether resource exist in store or not" This reverts commit 0f0ea9cbcc63e19d196e110ea2c34ef488586c99. * Revert "refactor: rename batch update to deploy based on discussion" This reverts commit bfe90293956416209b91e4f6ce8c2532c0604bfe. * Revert "refactor: redesign resource status" This reverts commit e863905d7aed16a68c0601965ddb4dd1022dabce. * refactor: update available statuses based on the latest convention * fix: update status transition based on the latest convention * refactor: rename batch update to deploy based on the latest discussion * refactor: re-implement resource create service based on the latest state transition * test: add test for backup * fix: lint errors * fix: resource manager to adopt the latest status transition Co-authored-by: Sandeep Bhardwaj <[email protected]> * fix: catch possible intermittent error during getting existing resource * fix: use custom not found error if encountered db not found error * fix: update resource service based on the latest state transition * fix: use float64 for proto values and not int * test: add missing tests * refactor: check resource status on deploy * test: add test * fix: fix lint issues * fix: fix flaky test * fix: fix issues while qa * fix: trim the resource name before sending * rework: rework the location of resources * fix: fix error message * fix: fix failing test * refactor: get urn from datastore * refactor: update urn in store * refactor: use resources kind as string * refactor: delete bigquery adapter * refactor: remove references to datastore * refactor: resource bounded context (#650) * wip: add domain models for resource bounded context * refactor: add handle and service for resources * refactor: add validate for resources * refactor: add converts for resource * refactor: update resource api * refactor: add test for resource handler * test: add test for metadata and dataset * test: add test for resource models * test: add test for resource * refactor: add resource test and fix lint issues * feat: add resource service * refactor: rename methods in resource repository for better clarity * feat: add batch update on resource service * refactor: simplify interface being used by service and update status * refactor: aedd extra config to resources * refactor: add resource manager * refactor: add bigquery handler for dataset, view and ext_table * refactor: define all methods on pointer receiver * feat: add resource replacement migration * feat: add function to convert string to status * feat: add resource repository * refactor: add table apis * refactor: add bigquery store * fix: fix tests for resources * refactor: unit test for resource repository to be independent from each other * test: refactor resource unit test to cover more cases * test: refactor service unit test for better readability * refactor: add exists check on resource handle * refactor: add batch update and multi error * feat: add index on resource using project namd and namespace name * feat: add tenant checks before creating resource * refactor: finish the code for batch update * fix: old resource usage error in backup repository * fix: fix failing test * test: update repository test to use oen parallelism * refactor: rename bigquery store from BigQuery -> Bigquery Co-authored-by: Anwar Hidayat <[email protected]> * refactor: resource handle accepts only the resource Co-authored-by: Anwar Hidayat <[email protected]> * refactor: wire the dependencies for resource handler Co-authored-by: Anwar Hidayat <[email protected]> * feat: add multi-error converter Co-authored-by: Sandeep Bhardwaj <[email protected]> * fix: add multi-to error converter Co-authored-by: Sandeep Bhardwaj <[email protected]> * refactor: update repository to allow for partial update and add unit test Co-authored-by: Sandeep Bhardwaj <[email protected]> * refactor: update the check for status * test: add test for resource manager * test: add test for resource models * fix: bug cannot find an existing secret Co-authored-by: Sandeep Bhardwaj <[email protected]> * fix: batch update behavior to create or update based on the provided status * feat: add telemetry for key resource implementations * fix: fix issues with batch update * test: add test for batch * fix: fix lint errors * refactor: remove unused function * refactor: add logger wrapper for server to simplify log * feat: add process logging in resource service * feat: add process logging in resource manager * fix: add log writer for resource service and handler * chore: change log level from warning to error for resource deploy in handler * refactor: Revert "feat: add telemetry for key resource implementations" This reverts commit e416fa6a5d63aa1d57e8ee78c1659658dd5e6cd5. * refactor: add some status mutation methods for resource * test: update unit test for mark status to use resource creation from existing Retaining the usage of function 'FromExisting' is necessary. One might argue that this is not a really good approach. Yes, in a certain cases it is not a good approach. An example in which we no longer use this function is when we are doing resource deployment, which requires update status. However, there are certain cases where we still need this function as a way for shortcut in setting status. An obvious example is when we are fetching resource in the database. The way we store it is in a string format, thus we need to convert it. Nonetheless, we don't know what status it is. And during the resource initialiation, it is initialized with 'unknown' status. In order to 'reach' the status from database, we have to perform some actions on marking statuses so it will become the same. This is tricky, and even the tricky part is we don't know how many steps we have to perform. As a side note, we should limit its usage as much as possible. * fix: remove resource status from equality factor * refactor: remove log writer usage in service layer * fix: allow reading error from repository when batch updating * refactor: restructure how to send response through log writer by prioritizing the success, skipped, error, and the errors * refactor: add logger other than writer when writing resource status * feat: add metric for batch update on skipped, success, and failure resources * fix: resource equality bug on tenant * fix: bug on empty resources for store batch update * fix: database migration conflict * fix: bug on skipped resource regardless of the status * feat: use complete message when encountering error during resource deployment * lint: fix linter issue * fix: re-order status to address additional cases * fix: make sure the resources to be processed have success validation status * fix: syntax and unit test error Co-authored-by: Sandeep Bhardwaj <[email protected]> * refactor: redesign resource status note: this commit still contains both syntactical and logical error * refactor: rename batch update to deploy based on discussion note: this commit still contains both syntactical and logical error * refactor: update resource manage to check whether resource exist in store or not note: this commit contains both syntactical and logical error * feat: add store existence status in resource * refactor: add update status on repo for resource in service note: this commit contains both syntactical and logical error * fix: add create failure and update failure status precedence for transition * refactor: re-implement service for create resource * refactor: update how status is changed through the transition for better readbility and maintenance * refactor: modify create resource to return multi error * fix: allows status transition to the current status * refactor: re-implement service for update resource * fix: add resource status in store * refactor: remove state transition management to simplify this removal is because the state transition changes can move to any status. concrete example of this is status 'unknown'. this status can change to any status based on the current service requirements. so, for now it will be removed. * fix: status bug on create and update resource * refactor: re-implement service for deploy resources * fix: status transition for repository * feat: add existence check on resource manager * feat: add existence check on bigquery datastore * fix: syntax error caused by missed contract * fix: linter issue in service * Revert "fix: linter issue in service" This reverts commit 56de127a5c47c0fdf5b231d5fdde3955340c939e. * Revert "fix: syntax error caused by missed contract" This reverts commit f970be1d84acd28d1a273c348468752d3fcc2da0. * Revert "feat: add existence check on bigquery datastore" This reverts commit 12ed8020854a9842a890c9af1e8e49b7384c15d8. * Revert "feat: add existence check on resource manager" This reverts commit 0cb92aa308d203d1bbd375e775d417e826de6448. * Revert "fix: status transition for repository" This reverts commit 4c0fa25f2deb1ff735022bb9fc86c571ee92bc4d. * Revert "refactor: re-implement service for deploy resources" This reverts commit 084f316b3b731389e2a1291383c02094d90d6905. * Revert "fix: status bug on create and update resource" This reverts commit cc3111af88549b8e2b3eed61e7fc429bbc7cf7cb. * Revert "refactor: remove state transition management to simplify" This reverts commit 697488bd5a2639237f8909b260a374e77b8520a9. * Revert "fix: add resource status in store" This reverts commit 75e2a45bc60fcb9139780c7ce11cff40e502076c. * Revert "refactor: re-implement service for update resource" This reverts commit 9c163df4477823d0f02d726c7989d02e5cc7ca74. * Revert "fix: allows status transition to the current status" This reverts commit 1fb76e610258c6d0ed0497af8a6b9c4514ca77ed. * Revert "refactor: modify create resource to return multi error" This reverts commit 6ad9634d525c29126bc9693150fc3e9c9091b7e4. * Revert "refactor: update how status is changed through the transition for better readbility and maintenance" This reverts commit 90aa2ede335c4f068d1ac0599f072b0107f94533. * Revert "refactor: re-implement service for create resource" This reverts commit f4bc3840441b7364f533c952a5d656a6698ccaee. * Revert "fix: add create failure and update failure status precedence for transition" This reverts commit 77eb5bba6758c109f89ff54441bd38e693012140. * Revert "refactor: add update status on repo for resource in service" This reverts commit 6970b32878af7cc02a538dac127b53de9b7fd573. * Revert "feat: add store existence status in resource" This reverts commit cd9520df8e0cf7feed2b0817cf8be1fa15929138. * Revert "refactor: update resource manage to check whether resource exist in store or not" This reverts commit 0f0ea9cbcc63e19d196e110ea2c34ef488586c99. * Revert "refactor: rename batch update to deploy based on discussion" This reverts commit bfe90293956416209b91e4f6ce8c2532c0604bfe. * Revert "refactor: redesign resource status" This reverts commit e863905d7aed16a68c0601965ddb4dd1022dabce. * refactor: update available statuses based on the latest convention * fix: update status transition based on the latest convention * refactor: rename batch update to deploy based on the latest discussion * refactor: re-implement resource create service based on the latest state transition * fix: resource manager to adopt the latest status transition Co-authored-by: Sandeep Bhardwaj <[email protected]> * fix: catch possible intermittent error during getting existing resource * fix: use custom not found error if encountered db not found error * fix: update resource service based on the latest state transition * fix: use float64 for proto values and not int * refactor: check resource status on deploy * test: add test * fix: fix lint issues * fix: fix flaky test * rework: rework the location of resources * refactor: get urn from datastore * refactor: update urn in store * refactor: use resources kind as string Co-authored-by: Sandeep Bhardwaj <[email protected]> * chore: update proton commit Co-authored-by: Anwar Hidayat <[email protected]> Co-authored-by: Anwar Hidayat <[email protected]> * feat: scheduling bounded context (#663) * refactor: create empty files for structure * refactor: import existing window code * fix: use domain errors in window * chore: add information about apis * refactor: mark api for removal * chore: add todo * fix: added coments * refactor: remove code not required * refactor: add job run input handler * refactor: add service for job run input * refactor: add new template engine and template context builder * fix: update proton hash * fix: update proton hash * refactor: add job run input compiler * fix: job run state handler * WIP * refactor: add dag compile for airflow * refactor: update dag template to use hook groups * refactor: add job value object * refactor: fix some issues related to dag compilation * refactor: add bucket factory * refactor: add airflow client and job run status * Fix: WIP * refactor: add airflow service * refactor: simplify the job model * refactor: change job model structure * refactor: add job details in job repo * fix: job_run operator states * refactor: align template and the expected dag * fix * fix: fix the compilation of job_run handler test * fix: WIP * fix: priority resolution test * fix: finish upload to scheduler * eip * fix: add job_run repo layer * fix: cleanup job run repo layer * wip * fix : linter issues * fix: test cases * fix: lint issues * fix: add dependency injection Scheduler bounded context * fix: increase test cases code coverage * test: add executor info test * fix: fix test * fix: add test cases * merge: pull master * fix: failing test case * refactor: update lib.py and dag template after merge with main * fix: increase code coverage * fix: add test cases * fix: add test cases * fix: add test cases * fix: make scheduler bounded context code coverage 97.6 percent * fix: add migration for scheduler bounded context * fix: update the scheduler callbacks * fix: update proton hash * fix: update proton hash * fix: test cases and linter issues * fix: update proton hash, fix failure type event name * fix: add back removed test * fix: test case * fix: add migrations Co-authored-by: Sandeep Bhardwaj <[email protected]> * refactor: update bigquery library (#677) * update: update bigquery library * fix: fix lint issues * feat: add job bounded context (#673) * feat: store job spec from AddJobSpecification Co-authored-by: sravankorumilli <[email protected]> * feat: store job resource from AddJobSpecification Co-authored-by: sravankorumilli <[email protected]> * feat: add dependency resolver and modify job repository Co-authored-by: sravankorumilli <[email protected]> * feat: do not break the AddJobSpec process when one of the job has failure Co-authored-by: sravankorumilli <[email protected]> * feat: implement resolvers and repositories, and complete flow for add job specification Co-authored-by: sravankorumilli <[email protected]> * feat: store additional dependency infos and fix AddJobSpecification flow based on scenarios Co-authored-by: sravankorumilli <[email protected]> * feat: remove deployment manager from job bounded context Co-authored-by: sravankorumilli <[email protected]> * chore: tidy go module * refactor: change job spec into spec to avoid stutter * refactor: rename dependency to upstream * refactor: move out job spec implementations to different files * refactor: Revert "refactor: move out job spec implementations to different files" This reverts commit f6c0c638c0fdb01bc37201e0d05d2a64a556bfbd. * fix: syntax and migration error after merge from main * refactor: rename dependency to upstream for implementations under core and ext packages * refactor: rename dependency into upstream for implementations under internal package * refactor: use only one error for better readability * refactor: remove unnecessary window implementation * refactor: use builder pattern for spec building for better readability * refactor: move out labels from one of required fields on job spec * feat: store job spec from AddJobSpecification Co-authored-by: sravankorumilli <[email protected]> * feat: store job resource from AddJobSpecification Co-authored-by: sravankorumilli <[email protected]> * feat: add dependency resolver and modify job repository Co-authored-by: sravankorumilli <[email protected]> * feat: do not break the AddJobSpec process when one of the job has failure Co-authored-by: sravankorumilli <[email protected]> * feat: implement resolvers and repositories, and complete flow for add job specification Co-authored-by: sravankorumilli <[email protected]> * feat: store additional dependency infos and fix AddJobSpecification flow based on scenarios Co-authored-by: sravankorumilli <[email protected]> * feat: remove deployment manager from job bounded context Co-authored-by: sravankorumilli <[email protected]> * chore: tidy go module * refactor: change job spec into spec to avoid stutter * refactor: rename dependency to upstream * refactor: move out job spec implementations to different files * refactor: Revert "refactor: move out job spec implementations to different files" This reverts commit f6c0c638c0fdb01bc37201e0d05d2a64a556bfbd. * fix: syntax and migration error after merge from main * refactor: rename dependency to upstream for implementations under core and ext packages * refactor: rename dependency into upstream for implementations under internal package * refactor: use only one error for better readability * refactor: remove unnecessary window implementation * refactor: use builder pattern for spec building for better readability * chore: deprecate deployment ID from AddJobSpecificationsResponse * fix: remove deprecated base unit plugin from tests * refactor: move tenant from job spec entity to job entity * refactor: use strongly typed data for fields under spec * chore: add todo for further development Co-authored-by: Arinda Arif <[email protected]> * refactor: move schedule interval to be optional * feat: implement delete job specification * feat: return created job names as part of AddJobSpecifications response * refactor: create job.ResourceURN for destination and job source * refactor: change upstream name type as job.Name * refactor: create job.SpecUpstreamName type and move complexity of project/job name to its behavior * test: fix test issue due to unimplemented mock post merging * feat: delete records in job_upstream when job is deleted * fix: issue job soft delete always runs after job hard delete * test: add test cases for delete job specification in handler and service * fix: alter job_id foreign key in job_run table and add foreign key in replay * chore: deprecate job source table * chore: fix lint issues in job BC * chore: remove job_id foreign key constraint to avoid issue before migration * refactor: add in jobservice to only return error and rearrange the methods in job service * chore: fix lint issue * fix: handle if optional job fields are empty before marshalling * chore: remove truncate job source in test setup * feat: implement Update in job repository * feat: implement Update in job service * feat: implement UpdateJobSpecifications in job handler * feat: add get job spec with filter on handler and service * feat: add placeholder for proto converter * feat: implement get by name and get all by project name * feat: bits on filter * feat: add test on filter job service * feat: implement adapter to convert orm to spec * test: add mock * refactor: add todo * test: add mock test for get and get all job service * feat: add get window * feat: implement ReplaceAll in job service * feat: initially implement Refresh in job service * fix: job repository adapter issues * feat: implement GetAllByTenant in job repository * feat: remove GetAllSpecsByTenant interface from JobService * feat: implement ReplaceAll in job handler * refactor: merge logic in GetAll getting jobs for refresh in Job Service * test: add get by job name test * feat: implement RefreshJobs in JobHandler * refactor: change Get and GetAll interface to return array of Job not Spec * refactor: add logs on job handler * fix: conflicts in JobService * test: add get all b project name test * test: add get all by resource destination test * feat: add replace all cli * refactor: adjust refresh cli with job bounded context handler * feat: add logs in job bounded context and fix test cases * test: add validate api * feat: add replace-all subcommand to job command * feat: register new job handler to optimus server * refactor: move constructing resource manager to external upstream resolver and fix test cases * chore: fix lint issues * test: add mock for job validate * test: add test cases for ReplaceAll and Refresh in handler * refactor: handle multiple namespaces in single refresh request * test: add validate api test * fix: metadata resource issue in job repository adapter * fix: remove plugin cleanup in replace all * fix: filter issue for string array type * feat: get info plugin * feat: add task builder * feat: add get job task info and refactor get service * feat: add get job with destination and sources details * refactor: remove recalculation of destination and sources * feat: introduce cyclic deps check on validate api * test: add test for validation api * test: add list job spec handler + add project name and namespace name on get job specifications * feat: adjust list job spec using existing filters * feat: refactor upstream resolvers to have both bulk and single job resolution method * feat: add job inspect in job bounded context * fix: issue on job handler post rebasing * feat: add projectName, namespaceName, taskName to job upstream model * feat: add task name and external field to job_upstream table * fix: issues on GetDownstreamByJobName and GetDownstreamByDestination * test: add get upstream, downstream, and new job to inspect test in job service * fix: issue when initializing upstream in resolver * feat: optimizing get job by source when building tree * refactor: add GetJobBasicInfo to return job info and logger for inspect case * feat: impement to job proto * test: improve test coverage filter * test: improve test coverage job get service * refactor: rename getall -> getbyfilter * fix: issue in job inspect when job spec and job name are empty, and implement test cases in job inspect handler * test: add more GetJobBasicInfo test cases * fix: converting to job proto issue and fix unit tests * refactor: remove ResourceURN constructor * test: add test for get by filters * test: add test for get by filters when namespace is emptu * test: add test for get task info * feat: handle case when job is soft deleted and wants to readd * feat: handle case when job is soft deleted and wants to update * feat: parallelize job bulk add and bulk update * feat: parallelize upstream resolver * feat: add and fix logs in job bounded context * fix: lint issues in job bounded context * refactor: remove upstream type check in upstream resolved constructor * fix: missing some internal upstream resolution issue * test: add test for get job specification handler * test: add test for get job specifications handler * test: add test for list job specs handler * test: add test for check job specifications handler * fix: issue unable to get active jobs in repository and add test cases * feat: exclude soft deleted jobs when querying from job table * test: add test for get job task handler * test: add test cases in job service * fix: issue when http upstream name is not returned in JobInspect and add test cases * test: add external upstream resolver test cases * test: fix intermittent test failure on job service * test: add plugin service test cases * chore: fix lint issues * fix: rename indexes of job_old to avoid duplication * refactor: better logging on spec validation * fix: allow empty value in map * refactor: remove unused updated_at and deleted_at fields in job_upstream table * feat: handle record not found error in GetByFilter * refactor: use internal error model in external upstream resolver * fix: upstream updated_at not found issue and record not found issue in job repository * fix: unresolved static upstream issue not shows up * fix: show root cause error when unable to insert upstream * fix: safely drop constraint in migration using if exists clause * fix: show proper log when static upstream is not resolved * fix: issue when the existing job is not considered equal with the same incoming job * fix: ambiguous logs in replace all * refactor: change log info on replace all * fix: inaccurate classifying job spec issue and show detailed logs * fix: empty unknown upstream in job inspect respond * fix: ambiguous error logs when running in bulk * fix: refresh needs to re-generate job and update as sources might be changed * fix: job refresh not refreshing job sources issue * fix: issue when job basic info is not found for inspecting * fix: add and move constraints to proper places * fix: handle null fields when adapting job specification * fix: job_run foreign key wrong add constraint statement * test: add GetJobNameWithInternalUpstreams to also return upstream from different projects * fix: insert job_id and job_upstream_id to job_upstream table * fix: get downstream full names should only return the active downstreams * fix: handle deletion failure and fix log messages * refactor: remove deployment id from add and refresh response * test: fix failed unit test in job handler * chore: fix lint issues * test: add job_upstream and job_old tables to the list of tables to drop * fix: missing proto models and variables * fix: adjust migration files after merge * chore: remove unused code as replaced by job bc * fix: invalid character in migration file 35 * test: add job and spec entity unit tests * chore: fix lint issue * test: add spec IsEqual test cases * chore: add more spec test cases and remove unused function in spec * refactor: call validate inside the build function for spec attributes * fix: doubled import issue * chore: add todos in job bc Co-authored-by: sravankorumilli <[email protected]> Co-authored-by: Anwar Hidayat <[email protected]> Co-authored-by: Dery Rahman Ahaddienata <[email protected]> * fix: port changes from 0.4 (#685) * fix: port changes from 0.4 branch * fix: delete unwanted files & fix the typos * Feature/scheduler upstream (#682) * fix: add scheduler upstream enrich logic * fix: upstream fetch logic * fix: undo function name change * fix: remove whitespace change * fix: bugs in the upload to scheduler flow * fix: remove model dependency * fix : update dag tamplate file * fix: error handling upload to scheduler * fix: handle missing end dates in dags * refactor: cleanup unused code (#680) * refactor: move writers to internal * refactor: move version handler, delete api * refactor: delete unused job module * refactor: delete unused service module * refactor: remove airflow2 module * refactor: remove mocks not required * fix: fix compilation issues * refactor: delete unused repository * refactor: delete unused resource manager * refactor: remove unused models * fix: remove resource test * refactor: remove unused mocks * refactor: add test for errors * fix: fix lint error * refactor: remove global plugin repo * revert: bring back plugin test * refactor: move models and mocks to internal * refactor: remove redundant code * chore: remove restructure md file * refactor: delete unused code * fix: fix build after merge * fix; resolve operator run repo query bugs * fix: take host from project config for internal dependencies * fix: job run service test include logger * fix: tests and revert config/build.go version change * fix: handle sla miss callback * refactor: job bounded context (#687) * refactor: remove dto.RawUpstream and use job.Upstream instead * refactor: rename alert and upstream in spec as alertSpec and upstreamSpec * feat: include resource urn in unknown upstream job inspect response * fix: issue when update job of another namespace simplify if statements in pre-check of updata and add * refactor: remove get downstream full names and replaced with get downstream by job name * refactor: refactor upstream resolvers * refactor: extract out jobProtos in job handler * refactor: move job filters creation from handler to service * refactor: rename jobNamesToSkip to jobNamesWithValidationError * refactor: remove bulkRefreshSource and use bulkUpdate in refresh * refactor: remove duplication of generate jobs in bulk * chore: fix lint issues * chore: add todos in job bc * fix: merge unresolved and internal upstream issue * test: add job entity test cases * chore: fix import issues post merging * fix: remove unnecessary wrapping in job config and asset * fix: discrepancy when constructing job from yaml and from db * fix: unresolved inferred upstream not shown in job inspect * refactor: move merge unresolved with resolved upstream to WithUpstreamList behavior * test: add value of upstream check after running replace upstream * test: add more job entity test cases * refactor: rename WithUpstreamList to WithUpstreams * fix: job service validate to use stream response writer * refactor: pass only task name to plugin service info and rename GetTaskInfo * refactor: reuse fromJobProtos in check job specifications handler * test: add more job spec and plugin service test cases * fix: remove gorm order by, use order by in raw query instead * fix: sensor logic fix * fix: job bounded context issues (#691) * fix: missing internal upstream issue * fix: job metadata not stored properly issue * fix: issue on replace all when tenant is not exist yet * chore: remove todo file * fix: remove operator state check, before new operator run creation * fix: test cases for operator runs Co-authored-by: Anwar Hidayat <[email protected]> Co-authored-by: Anwar Hidayat <[email protected]> Co-authored-by: Yash Bhardwaj <[email protected]> Co-authored-by: Arinda Arif <[email protected]> Co-authored-by: sravankorumilli <[email protected]> Co-authored-by: Dery Rahman Ahaddienata <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is to address this issue. Changes on this PR:
Note: