-
Notifications
You must be signed in to change notification settings - Fork 22
Updating of artifacts and artifact data #83
Updating of artifacts and artifact data #83
Conversation
Codecov Report
@@ Coverage Diff @@
## master #83 +/- ##
==========================================
+ Coverage 68.70% 69.54% +0.84%
==========================================
Files 31 31
Lines 1262 1376 +114
==========================================
+ Hits 867 957 +90
- Misses 325 341 +16
- Partials 70 78 +8
Flags with carried forward coverage won't be shown. Click here to find out more.
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
Thank you for opening this pull request! 🙌 |
Signed-off-by: Nick Müller <[email protected]>
Currently only supports replacement of associated ArtifactData Uses unreleased flyteidl update for new datacatalog endpoint Signed-off-by: Nick Müller <[email protected]>
Updated Dockerfile to be in line with flyteadmin Updated GitHub automation to use Go 1.18 Signed-off-by: Nick Müller <[email protected]>
Signed-off-by: Nick Müller <[email protected]>
Signed-off-by: Nick Müller <[email protected]>
Signed-off-by: Nick Müller <[email protected]>
Signed-off-by: Nick Müller <[email protected]>
Signed-off-by: Nick Müller <[email protected]>
637a9dd
to
f3391f1
Compare
Signed-off-by: Nick Müller <[email protected]>
Prevents DB entries without underlying blob storage data in case of a partial update failure Signed-off-by: Nick Müller <[email protected]>
logger.Debugf(ctx, "Get artifact by id %v", request.GetArtifactId()) | ||
artifactKey := transformers.ToArtifactKey(datasetID, request.GetArtifactId()) | ||
|
||
key := queryHandle.GetArtifactId() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there any reason we changed this from using a switch over to len(queryHandle.GetArtifcatId())
from request.QueryHandle.(type)
. Was it required because of the addition of UpdateArtifactRequest handling?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@hamersaw Yes, exactly. Unfortunately the type generated for the UpdateArtifactRequest
is a datacatalog.UpdateArtifactRequest_ArtifactId
(instead of a datacatalog.GetArtifactRequest_ArtifactId
), however both implement GetArtifactId() string
.
As the oneof
interface is not exported in the generated protobuf definitions, we can't use a shared interface, so this new artifactQueryHandle
interface was the easiest I came up with (which allows both to be passed as a single method parameter).
Congrats on merging your first pull request! 🎉 |
* Fixed repo interface mocks not being generated Signed-off-by: Nick Müller <[email protected]> * Implemented updating of artifacts Currently only supports replacement of associated ArtifactData Uses unreleased flyteidl update for new datacatalog endpoint Signed-off-by: Nick Müller <[email protected]> * Updated project to Go 1.18 Updated Dockerfile to be in line with flyteadmin Updated GitHub automation to use Go 1.18 Signed-off-by: Nick Müller <[email protected]> * Fixed artifact data upsert in Artifact.Update Signed-off-by: Nick Müller <[email protected]> * Removed golang_dockerfile from boilerplate update Signed-off-by: Nick Müller <[email protected]> * UpdateArtifact returns ID of artifact updated Signed-off-by: Nick Müller <[email protected]> * Updated to latest version of flytepropeller Signed-off-by: Nick Müller <[email protected]> * Updated to latest released version of flyteidl Signed-off-by: Nick Müller <[email protected]> * Updated to latest released flytestdlib version Signed-off-by: Nick Müller <[email protected]> * Missing artifact data is now removed after DB models have been updated Prevents DB entries without underlying blob storage data in case of a partial update failure Signed-off-by: Nick Müller <[email protected]> Signed-off-by: Nick Müller <[email protected]>
TL;DR
This PR extends datacatalog to support the updating of artifacts and overwriting of stored artifact data as well as some minor cleanup/housekeeping.
Type
Are all requirements met?
Complete description
The new
UpdateArtifact
endpoints allows for an existing artifact to be updated. At the moment, only the underlyingArtifactData
can be overwritten.While updating, datacatalog will create any new data keys, overwrite any existing ones with the new data provided and remove no longer provided elements from the underlying blob storage. To facilitate a full cleanup, deletion of artifact data was implemented as well.
This PR also fixes the mock generation for the repository interfaces as they seem to have never been re-generated using the latest
mockery
version.Additionally, the datacatalog project was moved to Go 1.18 and the Dockerfile setup was adapted to be in line with flyteadmin. Some minor cleanup/housekeeping was performed to make the linter happy.
As this uses a currently unmerged
flyteidl
version, the PR is created as a draft (for review) until a new version offlyteidl
is available.Tracking Issue
flyteorg/flyte#2867
Follow-up issue
NA