Storage Browser Default Auth #13866

ashika112 · 2024-09-30T21:39:09Z

Description of changes

Creates Amplify Auth adapter [This is expected to be moved to UI. Including this atm to make testing easier.]
Creates ListPaths which is equivalent to ListLocations
Updates type for ListPaths

TODO

Pagination + Memoization
Aligning behavior on user group edge case

Description of how you validated changes

Unit test
Manual testing using sample app

Checklist

PR description included
yarn test passes
Unit Tests are changed or added
Relevant documentation is changed or added (and PR referenced)

Checklist for repo maintainers

Verify E2E tests for existing workflows are working as expected or add E2E tests for newly added workflows
New source file paths included in this PR have been added to CODEOWNERS, if appropriate

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

ashika112 · 2024-10-04T19:32:29Z

packages/storage/__tests__/internals/amplifyAuthAdapter/createAmplifyAuthConfigAdapter.test.ts

Adapter will most likely be moved to UI

ashika112 · 2024-10-04T19:43:55Z

packages/storage/src/internals/amplifyAuthConfigAdapter/createAmplifyAuthConfigAdapter.ts

Adapter will most likely be moved to UI

packages/storage/src/internals/amplifyAuthConfigAdapter/createAmplifyListLocationsHandler.ts

packages/storage/src/internals/types/common.ts

packages/storage/src/internals/types/credentials.ts

packages/storage/src/internals/amplifyAuthConfigAdapter/generateLocationsFromPaths.ts

…plify-js into feat/sb-default-auth

ashika112 · 2024-10-16T21:39:56Z

packages/storage/src/internals/types/credentials.ts

+/**
+ * @internal
+ */
+export interface PathAccess {


Keeping this separate for now. In a separate PR will align this and managed Auth Location access into one.

AllanZhengYP

Looks good to me! Only have 1 nit comment.

AllanZhengYP · 2024-10-17T00:41:02Z

packages/core/src/singleton/AmplifyOutputs/types.ts

@@ -50,6 +50,8 @@ export interface AmplifyOutputsStorageBucketProperties {
 	bucket_name: string;
 	/** Region for the bucket */
 	aws_region: string;
+	/** Paths to object with access permissions */
+	paths?: Record<string, Record<string, string[] | undefined>>;


The in-line type is a bit hard the follow, what about using a interface with property names:

type Permission = string; interface StorageBucketPaths { [pathPrefix: string]: { [accessType: string]: Permission[] | undefined; } }

Our ESLint does not allow for index signature. But i think we can create alias and align if we want. I can look into this in my follow up PR for pagination if there are no other comments :)

packages/storage/src/internals/amplifyAuthConfigAdapter/createAmplifyAuthConfigAdapter.ts

packages/storage/src/internals/types/credentials.ts

* chore: enable storage-browser preid release * chore: sync main (#13478) * release(required): Parsing custom oAuth in amplify_outputs (#13474) * update parseAmplify logic * revert custom oAuth from gen1 config * update bundle size * chore(release): Publish [skip release] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - [email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - [email protected] * chore(release): update API docs [skip release] --------- Co-authored-by: ashika112 <[email protected]> Co-authored-by: aws-amplify-bot <[email protected]> Co-authored-by: Jim Blanchard <[email protected]> * feat(storage): add delimiter support (#13480) * feat: add delimiter input/output types * feat: pass Delimiter parameter to ListObjectsV2 API * chore: add unit tests * chore: bump bunde size * chore: address feedback * chore: fix build * chore: address feedback * chore: address feedback * chore: address feedback * chore: enable storage-browser preid release (#13524) chore: fix npm dist tag to be storage-browser * feat(storage): add base types for storage browser (#13528) * feat(storage): add base types * fix(storage): address feedbacks Co-authored-by: israx <[email protected]> * feat(storage): add creds store scaffolding and update types (#13558) * feat(storage): add cred store lru implementation (#13561) * refactor(storage): decouple utils from Amplify singleton (#13562) * feat: add config constructor * refactor: remove singleton reference from storage utils * refactor: update storage utils * chore: update upload api * chore: address feedback * chore: fix unit tests * chore: remove singleton reference * chore: add license headers * chore: address feedback * chore: update bundle size * chore: address feedback * chore: update bundle size * feat(storage): add cred store creation implementation (#13575) Co-authored-by: israx <[email protected]> * feat(storage): use WeakMap for store registry (#13586) * feat(storage): use WeakMap for store registry * chore(storage): export storage browser utils from @aws-amplify/storage/storage-browser * doc(storage): add disclaimer * feat(storage): Added getDataAccess & listCallerAccessGrant clients (#13582) * feat(storage): add an adapter interface for storage browser (#13576) --------- Co-authored-by: Jim Blanchard <[email protected]> * Revert "refactor(storage): decouple utils from Amplify singleton (#13562)" (#13597) This reverts commit 1079e7f. * feat(storage): simplify the location cred provider option input (#13601) Remove the un-essetnial validation of per-API's location credentials provider input scope and permission for now. * feat: Implement getLocationCredentials handler & integrate with adapter (#13600) * feat(storage): implement listLocations API and creation handler (#13602) * chore: expose path storage-browser from scoped package (#13611) chore: expose path storage-browser from scoped package Co-authored-by: Ashwin Kumar <[email protected]> * feat(storage): enables location credentials provider (#13605) * feat: add location credentials provider * chore: add unit tests * chore: address feedback * chore: add locationCredentialsOption to copy * chore: remove casting types * chore: assert idenitity id * chore: avoid export common options interface * chore: address feedback * chore: fix test * chore: address feedback * address feedback * chore: clean-up types * chore: add test * chore: update api bundlesize * feat(storage): resolve merge issue with multibucket * chore: update bundle size for config change and s3 multibucket * chore: address feedbacks * feat(storage): introduce preventOverwrite option to uploadData via HeadObject (#13640) * feat(storage): introduce preventOverwrite operation to uploadData via HeadObject * fix: add missing license and remove dependency on core in preventOverwrite validator * chore: update storage:uploadData bundle size * feat: move existing object validation to before completeMultipartUpload * fix: increase storage:uploadData bundle size * fix(storage): export storage-browser types for TS v4.2+ (#13647) * chore(storage-browser): export store and credentials related types, update createListLocationsHandler (#13660) * feat(storage): support force refresh location credentials (#13589) * feat(storage): require temporary creds for storage browser interfaces (#13664) * feat: introduce CRC32 checksums to storage:uploadData API (#13649) Co-authored-by: Donny Wu <[email protected]> * fix(storage-browser): listCallerAccessGrantsDeserializer not parsing multiple AccessGrant instances (#13671) * fix(storage-browser): listCallerAccessGrantsDeserializernot parsing multiple AccessGrant instances * chore: add unit tests for single and multiple grants --------- Co-authored-by: Ashwin Kumar <[email protected]> * chore(storage-browser): expose additional input output types (#13682) * chore(storage-browser): expose additional internal types * address feedback * remove 'applicationArn' from listCallerAccessGrant unit test * Update packages/storage/src/storageBrowser/apis/listCallerAccessGrants.ts Co-authored-by: Caleb Pollman <[email protected]> --------- Co-authored-by: Ashwin Kumar <[email protected]> Co-authored-by: Caleb Pollman <[email protected]> * chore: add ui to storage browser co-owner * chore: update bundle size * chore: enable storage-browser integ test (#13698) Co-authored-by: Ashwin Kumar <[email protected]> * chore(storage): update s3 control model (#13705) * chore(storage): update s3 control model * fix: move permission validation to custom client * chore: formatting code * chore(storage-browser): pin crc-32 dep at 1.2.2 (#13752) * chore(storage-browser): pin crc-32 dep at 1.2.2 * chore: update lock file * chore(storage): add durability check for urls in put object (#13746) chore: add durability check for presigned urls in put object * fix(storage-browser): missing error wrapping for s3 control responses (#13779) * feat: validate corroborating elements in response, and compare echoed elements between request/response (#13764) * feat: add list call response validation * fix: add input key to test mock * feat: align list response validations with design * fix: increment bundle size for storage:list --------- Co-authored-by: Donny Wu <[email protected]> * feat(storage): add support for conditional headers to copy, and validate serialization (#13772) * feat: add notModifiedSince and eTag to copy * missing tests for durability helpers * add tests for integrity helpers * feat: add URL validation and tests for copyObject * chore: increase bundle size of storage:copy * feat: clean-up copy header validation logic * fix: revert copy option interface name changes --------- Co-authored-by: Donny Wu <[email protected]> * feat: adding object url checks (#13810) * chore: durability checks for create & complete multipart (#13809) * chore: durability checks for create & complete multipart * fix: parsePayload mock path * chore: enable e2e and tagged release (#13848) * chore: enable e2e and tagged release * core: update tag name * Revert "core: update tag name" This reverts commit c994f51. --------- Co-authored-by: Ashwin Kumar <[email protected]> * chore: Setup storage internals route (#13858) * chore: increse bundle size * chore: Refactor contents of `storage-browser/` path into `internals/` (#13859) * feat(storage): export Default Part Size constant (#13851) * chore: adds a flag to uploadData api output * chore: update comments for the flag * chore: revert some naming changes * chore: revert and make internal paramter optional with a default value * chore: update bundle size * chore: update to expose constant instead of bool in return type * chore: update export tests * chore: increase bundle size * chore: Setup storage internals route (#13858) * chore: Refactor contents of `storage-browser/` path into `internals/` (#13859) * fix: multipart upload storage crc32 integrity (#13878) * fix: multipart upload storage crc32 integrity * address comments Co-authored-by: AllanZhengYP <[email protected]> * fix linter --------- Co-authored-by: Ashwin Kumar <[email protected]> Co-authored-by: AllanZhengYP <[email protected]> * feat(storage): internal GetProperties API (#13869) --------- Co-authored-by: Jim Blanchard <[email protected]> * feat(storage): add new internal remove api (#13880) feat(internal-remove): add new internal remove api Co-authored-by: Ashwin Kumar <[email protected]> * chore: Add internal getUrl API (#13882) * feat(storage): internals list API (#13874) * feat(storage): add new internal downloadData api (#13887) * feat(internal-remove): add new internal downloadData api * code cleanup * code cleanup * chore: fix ts doc --------- Co-authored-by: Ashwin Kumar <[email protected]> * feat: optional checksum algorithm for upload (#13849) * feat: opt in checksum * fix: revert local prettier suggestion * fix: up size limit for storage upload data * feat: react native crc32 * fix: up bundle size limit and fix typo * feat: add documentation for checksumAlgorithm * fix: update bundle size limit * fix: update bundle size limit * fix: address pr feedbacks * fix: bundle-size limit --------- Co-authored-by: AllanZhengYP <[email protected]> * feat(storage): add remaining copy changes for internals (#13889) * chore: add remaining copy changes for internals * chore; explicitly call internal API with params * chore: copy return type * chore: feedback changes * feat(storage): internal uploadData implementation (#13888) * feat(storage): internal uploadData implementation * chore: update bundle size * fix: address feedbacks * revert: optional checksum algorithm for upload (#13849) (#13910) This reverts commit 02cb08a. * fix(storage): internals list function not able to decide which output types to use (#13915) chore: add function overloading to list * feat(storage): Add API support for Expected Bucket Owner (#13914) * Update Top/Internal API for expected bucket owner feat --------- Co-authored-by: JoonWon Choi <[email protected]> * chore: sync main with storage browser integrity branch (#13928) * chore(release): Set core metadata [skip release] * chore(release): Publish [skip release] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - [email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - [email protected] * chore(release): Update API docs [skip release] * chore: update contributing guide vite troubleshooting (#13826) * docs: update contributing guide vite troubleshooting * Update CONTRIBUTING.md --------- Co-authored-by: AllanZhengYP <[email protected]> * chore(ai): add UpdateConversation enum (#13920) * chore(release): Set core metadata [skip release] * chore(release): Publish [skip release] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - [email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - @aws-amplify/[email protected] - [email protected] * chore(release): Update API docs [skip release] * fix(storage): multipart upload is firing on 0 bytes data (#13927) * chore: update bundle size --------- Co-authored-by: erinleigh90 <[email protected]> Co-authored-by: aws-amplify-bot <[email protected]> Co-authored-by: Parker Scanlon <[email protected]> Co-authored-by: AllanZhengYP <[email protected]> Co-authored-by: Danny Banks <[email protected]> Co-authored-by: ashika112 <[email protected]> Co-authored-by: Hui Zhao <[email protected]> * feat: Add expectedBucketOwner to remaining internal APIs (#13932) feat: Add expectedBucketOwner to remaining internal APIs. * feat(storage): add advanced option to disable upload cache (#13931) * feat(storage): set allowedByApp to listCallerAccessGrants (#13934) * Storage Browser Default Auth (#13866) * first draft poc * upadtes * add listPaths API * update new file structure * fix types * refactor types and utils * update tests * fix test * fix bundle size test * update the listLocation handler * rename util * update Path type * fix missed type * chore(ci): add new storage-gen2-internal e2e test (#13916) Co-authored-by: Ashwin Kumar <[email protected]> * chore(storage): remove credential store and managed auth config factory (#13944) * feat: optional checksum algorithm for upload (#13939) Co-authored-by: AllanZhengYP <[email protected]> * feat: preventOverwrite with if-none-match header (#13954) Co-authored-by: AllanZhengYP <[email protected]> * Feat: Add pagination to Amplify Default Auth storage Browser (#13897) * update the listLocation handler * implement memoization * add pagination logic * update usergroup logic & test * update getPaginated Locations * fix failing test * test(storage): refactor unit tests for public & internal facade (#13955) * Chore: Remove createAmplifyAdapter & refactor (#13974) remove createAmplifyAdapter & refactor * update bundle size * fix bundle size & test * Fix: retry failure in storage retryDecider (#13977) fix retry failure * feat: validate uploaded parts before completing upload (#13763) Co-authored-by: Donny Wu <[email protected]> Co-authored-by: Allan Zheng <[email protected]> * feat(storage): add customEndpoint to internal apis in advanced options (#13961) * feat: add baseEndpoint to advanced options * feat: add baseEndpoint to customEndpoint * feat: thread baseEndpoint through resolved config to endpoint resolver * add customEndpoint advanced option to internals storage data-plane apis * add customEndpoint advanced option to internals storage control-plane apis * fix unit test * code cleanup * increase bundle size * wire up customEndpoint on copy API * increase the bundle size * add customEndpoint unit tests for all data and control apis * increase bundle size * update ts docs * add additional error unit tests for endpointResolver * add unit tests for internals/ apis * code cleanup * address feedback * add comment for ForcePathStyleEndpointNotSupported ErrorCode * increase bundle size * remove docs links from error recovery message --------- Co-authored-by: Erin Beal <[email protected]> Co-authored-by: Ashwin Kumar <[email protected]> * fix(core): support endpoint resolver accepting both input and config(#13985) * feat(storage): allow checksum algo for internal upload API (#14002) * feat: allow setting encoding type for list calls * refactor: move to list internal * fix: add license * fix(storage): bug in copy unmodified since to use UTC (#14025) fix to use UTC and test * Revert "chore: enable e2e and tagged release (#13848)" This reverts commit 3dfde04. * revert tagged release --------- Co-authored-by: Allan Zheng <[email protected]> Co-authored-by: israx <[email protected]> Co-authored-by: ashika112 <[email protected]> Co-authored-by: aws-amplify-bot <[email protected]> Co-authored-by: Jim Blanchard <[email protected]> Co-authored-by: Ashwin Kumar <[email protected]> Co-authored-by: Ashwin Kumar <[email protected]> Co-authored-by: Jamie Epp <[email protected]> Co-authored-by: Jamie Epp <[email protected]> Co-authored-by: Caleb Pollman <[email protected]> Co-authored-by: Donny Wu <[email protected]> Co-authored-by: Caleb Pollman <[email protected]> Co-authored-by: Sergio Castillo Yrizales <[email protected]> Co-authored-by: Joon Choi <[email protected]> Co-authored-by: JoonWon Choi <[email protected]> Co-authored-by: erinleigh90 <[email protected]> Co-authored-by: Parker Scanlon <[email protected]> Co-authored-by: Danny Banks <[email protected]> Co-authored-by: Hui Zhao <[email protected]> Co-authored-by: ashika112 <[email protected]> Co-authored-by: Erin Beal <[email protected]> Co-authored-by: Pranav Malewadkar <[email protected]> Co-authored-by: Pranav Malewadkar <[email protected]>

ashika112 and others added 9 commits September 30, 2024 14:28

first draft poc

61ebc51

upadtes

a834fcc

add listPaths API

337068b

update new file structure

9d209e1

fix types

537275c

refactor types and utils

631dd7f

update tests

ee5dceb

fix test

70882a7

Merge branch 'storage-browser/integrity' into feat/sb-default-auth

6fdf40a

ashika112 marked this pull request as ready for review October 4, 2024 19:19

ashika112 requested review from haverchuck, cshfang, jimblanc, HuiSF and a team as code owners October 4, 2024 19:19

fix bundle size test

19468f2

ashika112 requested a review from a team as a code owner October 4, 2024 19:31

ashika112 commented Oct 4, 2024

View reviewed changes

update the listLocation handler

9cbc71d

ashika112 commented Oct 4, 2024

View reviewed changes

AllanZhengYP reviewed Oct 8, 2024

View reviewed changes

ashika112 added 4 commits October 16, 2024 14:18

rename util

beb1e2d

update Path type

4a7ed81

Merge branch 'storage-browser/integrity' of github.com:aws-amplify/am…

b5f7500

…plify-js into feat/sb-default-auth

fix missed type

4a063c7

ashika112 commented Oct 16, 2024

View reviewed changes

AllanZhengYP reviewed Oct 17, 2024

View reviewed changes

AllanZhengYP approved these changes Oct 17, 2024

View reviewed changes

calebpollman reviewed Oct 18, 2024

View reviewed changes

packages/storage/src/internals/amplifyAuthConfigAdapter/createAmplifyAuthConfigAdapter.ts Show resolved Hide resolved

calebpollman reviewed Oct 18, 2024

View reviewed changes

packages/storage/src/internals/types/credentials.ts Show resolved Hide resolved

calebpollman approved these changes Oct 18, 2024

View reviewed changes

ashika112 merged commit e515d24 into aws-amplify:storage-browser/integrity Oct 18, 2024
28 checks passed

ashika112 deleted the feat/sb-default-auth branch October 18, 2024 18:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Storage Browser Default Auth #13866

Storage Browser Default Auth #13866

ashika112 commented Sep 30, 2024 •

edited

Loading

ashika112 Oct 4, 2024 •

edited

Loading

ashika112 Oct 4, 2024

ashika112 Oct 16, 2024

AllanZhengYP left a comment

AllanZhengYP Oct 17, 2024

ashika112 Oct 17, 2024

Storage Browser Default Auth #13866

Storage Browser Default Auth #13866

Conversation

ashika112 commented Sep 30, 2024 • edited Loading

Description of changes

TODO

Description of how you validated changes

Checklist

Checklist for repo maintainers

ashika112 Oct 4, 2024 • edited Loading

Choose a reason for hiding this comment

ashika112 Oct 4, 2024

Choose a reason for hiding this comment

ashika112 Oct 16, 2024

Choose a reason for hiding this comment

AllanZhengYP left a comment

Choose a reason for hiding this comment

AllanZhengYP Oct 17, 2024

Choose a reason for hiding this comment

ashika112 Oct 17, 2024

Choose a reason for hiding this comment

ashika112 commented Sep 30, 2024 •

edited

Loading

ashika112 Oct 4, 2024 •

edited

Loading