Implement custom storage for orgs #2093
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tw4l
force-pushed
the
issue-578-custom-storage
branch
from
df5b6e9 to
10ab1b6
Compare
tw4l
force-pushed
the
issue-578-custom-storage
branch
9 times, most recently
from
f226271 to
eb065b6
Compare
This was referenced Oct 17, 2024
- Set access_endpoint_url to the endpoint url with bucket so that we can generate a presigned URL as expected - Make adding bucket in verify_storage_upload a backup routine after first exception is raised
Previously, files in a default bucket were prefixed with the oid but were not in custom storages. This commit removes that distinction to aid in copying files in buckets, removing the need for unnecessary filepath manipulation. The CopyBucketJob now only copies an organization's directory rather than the entire bucket to prevent accidentally copying another organization's data.
Creating a bucket in the verification stage for adding custom storages if it didn't exist was useful for testing but is an anti-pattern for production, so we remove it here.
tw4l
force-pushed
the
issue-578-custom-storage
branch
from
c5e88e3 to
a867411
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #578
Adds
Notes
Currently, no delete operations happen for a a bucket previously used as a primary or replica location that is unset. Files are copied to the new bucket to ensure there are no usability issues moving forward in the app, but the files are not automatically deleted from the source after the copy job. We could add that but I wonder if it's safer, especially in the early days of testing, to perform that cleanup manually as desired.
Once we're comfortable, we can change the rclone command in the
copy_job.yamlbackground job template fromcopytomoveif we want it to automatically clean up files from the source location on completion. Since the same template is used for copying files from an old primary storage to a new primary storage as well as to replicate from primary storage to a new replica location, we'd want to make sure the latter still usescopyso as not to delete files from the primary storage location.TODO