diff --git a/dev/api/index.html b/dev/api/index.html index b155e206..7c26a944 100644 --- a/dev/api/index.html +++ b/dev/api/index.html @@ -25,4 +25,4 @@ Requests.put(URI(url), "Hello!"; headers=Dict("Content-Type" => "text/plain"))

Permissions

source
AWSS3.s3_nuke_bucket — Function
s3_nuke_bucket([::AbstractAWSConfig], bucket_name)

Deletes a bucket including all of the object versions in that bucket. Users should not call this function unless they are certain they want to permanently delete all of the data that resides within this bucket.

The s3_nuke_bucket is purposefully not exported as a safe guard against accidental usage.

Warning

Permanent data loss will occur when using this function. Do not use this function unless you understand the risks. By using this function you accept all responsibility around any repercussions with the loss of this data.

API Calls

Permissions

source

S3Path

Note that S3Path implements the AbstractPath interface, some the FilePathsBase documentation for the interface here.

AWSS3.S3Path — Type
S3Path()
 S3Path(str::AbstractString; version::Union{Nothing, AbstractString}=nothing, config::Union{Nothing, AWS.AbstractAWSConfig}=nothing)
-S3Path(path::S3Path; isdirectory=path.isdirectory, version=path.version, config=path.config)

Construct a new AWS S3 path type which should be of the form "s3://<bucket>/prefix/to/my/object".

NOTES:

  • Directories are required to have a trailing / due to how S3 distinguishes files from folders, as internally they're just keys to objects.
  • Objects p"s3://bucket/a" and p"s3://bucket/a/b" can co-exist. If both of these objects exist listing the keys for p"s3://bucket/a" returns [p"s3://bucket/a"] while p"s3://bucket/a/" returns [p"s3://bucket/a/b"].
  • The drive property will return "s3://<bucket>"
  • On top of the standard path properties (e.g., segments, root, drive, separator), S3Paths also support bucket and key properties for your convenience.
  • If version argument is nothing, will return latest version of object. Version can be provided via either kwarg version or as suffix "?versionId=<object_version>" of str, e.g., "s3://<bucket>/prefix/to/my/object?versionId=<object_version>".
  • If config is left at its default value of nothing, then the latest global_aws_config() will be used in any operations involving the path. To "freeze" the config at construction time, explicitly pass an AbstractAWSConfig to the config keyword argument.
source
Base.stat — Function
stat(fp::S3Path)

Return the status struct for the S3 path analogously to stat for local directories.

Note that this cannot be used on a directory. This is because S3 is a pure key-value store and internally does not have a concept of directories. In some cases, a directory may actually be an empty file, in which case you should use s3_get_meta.

source
Base.Filesystem.mkdir — Function
mkdir(fp::S3Path; recursive=false, exist_ok=false)

Create an empty directory within an existing bucket for the S3 path fp. If recursive, this will create any previously non-existent directories which would contain fp. An error will be thrown if an object exists at fp unless exist_ok.

Note

Creating a directory in S3 creates a 0-byte object with a key set to the provided directory name.

source
Base.read — Function
read(fp::S3Path; byte_range=nothing)

Fetch data from the S3 path as a Vector{UInt8}. A subset of the object can be specified with byte_range which should be a contiguous integer range, e.g. 1:4.

source
AWSS3.get_config — Function
get_config(fp::S3Path)

Returns the AWS configuration used by the path fp. This can be stored within the path itself, but if not it will be fetched with AWS.global_aws_config().

source

Internal

AWSS3._s3_exists_dir — Function
_s3_exists_dir(aws::AbstractAWSConfig, bucket, path)

An internal function used by s3_exists.

Checks if the given directory exists within the bucket. Since S3 uses a flat structure, as opposed to being hierarchical like a file system, directories are actually just a collection of object keys which share a common prefix. S3 implements empty directories as 0-byte objects with keys ending with the delimiter.

It is possible to create non 0-byte objects with a key ending in the delimiter (e.g. s3_put(bucket, "abomination/", "If I cannot inspire love, I will cause fear!")) which the AWS console interprets as the directory "abmonination" containing the object "/".

source
AWSS3.s3_exists_versioned — Function
s3_exists_versioned([::AbstractAWSConfig], bucket, path, version)

Returns if an object version exists with the key path in the bucket.

Note that the AWS API's support for object versioning is quite limited and this check will involve try/catch logic. Prefer using s3_exists_unversioned where possible for more performant checks.

See s3_exists and s3_exists_unversioned.

API Calls

Permissions

  • s3:GetObjectVersion
  • s3:ListBucket (optional): allows requests to non-existent objects to throw a exception with HTTP status code 404 (Not Found) instead of HTTP status code 403 (Access Denied).
source
AWSS3.s3_exists_unversioned — Function
s3_exists_unversioned([::AbstractAWSConfig], bucket, path)

Returns a boolean whether an object exists at path in bucket.

See s3_exists and s3_exists_versioned.

API Calls

Permissions

  • s3:GetObject
  • s3:ListBucket (optional): allows requests to non-existent objects to throw a exception with HTTP status code 404 (Not Found) instead of HTTP status code 403 (Access Denied).
source
+S3Path(path::S3Path; isdirectory=path.isdirectory, version=path.version, config=path.config)

Construct a new AWS S3 path type which should be of the form "s3://<bucket>/prefix/to/my/object".

NOTES:

source
Base.stat — Function
stat(fp::S3Path)

Return the status struct for the S3 path analogously to stat for local directories.

Note that this cannot be used on a directory. This is because S3 is a pure key-value store and internally does not have a concept of directories. In some cases, a directory may actually be an empty file, in which case you should use s3_get_meta.

source
Base.Filesystem.mkdir — Function
mkdir(fp::S3Path; recursive=false, exist_ok=false)

Create an empty directory within an existing bucket for the S3 path fp. If recursive, this will create any previously non-existent directories which would contain fp. An error will be thrown if an object exists at fp unless exist_ok.

Note

Creating a directory in S3 creates a 0-byte object with a key set to the provided directory name.

source
Base.read — Function
read(fp::S3Path; byte_range=nothing)

Fetch data from the S3 path as a Vector{UInt8}. A subset of the object can be specified with byte_range which should be a contiguous integer range, e.g. 1:4.

source
AWSS3.get_config — Function
get_config(fp::S3Path)

Returns the AWS configuration used by the path fp. This can be stored within the path itself, but if not it will be fetched with AWS.global_aws_config().

source

Internal

AWSS3._s3_exists_dir — Function
_s3_exists_dir(aws::AbstractAWSConfig, bucket, path)

An internal function used by s3_exists.

Checks if the given directory exists within the bucket. Since S3 uses a flat structure, as opposed to being hierarchical like a file system, directories are actually just a collection of object keys which share a common prefix. S3 implements empty directories as 0-byte objects with keys ending with the delimiter.

It is possible to create non 0-byte objects with a key ending in the delimiter (e.g. s3_put(bucket, "abomination/", "If I cannot inspire love, I will cause fear!")) which the AWS console interprets as the directory "abmonination" containing the object "/".

source
AWSS3.s3_exists_versioned — Function
s3_exists_versioned([::AbstractAWSConfig], bucket, path, version)

Returns if an object version exists with the key path in the bucket.

Note that the AWS API's support for object versioning is quite limited and this check will involve try/catch logic. Prefer using s3_exists_unversioned where possible for more performant checks.

See s3_exists and s3_exists_unversioned.

API Calls

Permissions

  • s3:GetObjectVersion
  • s3:ListBucket (optional): allows requests to non-existent objects to throw a exception with HTTP status code 404 (Not Found) instead of HTTP status code 403 (Access Denied).
source
AWSS3.s3_exists_unversioned — Function
s3_exists_unversioned([::AbstractAWSConfig], bucket, path)

Returns a boolean whether an object exists at path in bucket.

See s3_exists and s3_exists_versioned.

API Calls

Permissions

  • s3:GetObject
  • s3:ListBucket (optional): allows requests to non-existent objects to throw a exception with HTTP status code 404 (Not Found) instead of HTTP status code 403 (Access Denied).
source
diff --git a/dev/index.html b/dev/index.html index 8e60a544..480291df 100644 --- a/dev/index.html +++ b/dev/index.html @@ -44,4 +44,4 @@ "this" julia> rm(file) # delete the file -UInt8[]
Warning

S3 is a pure key-value store, NOT a filesystem. Therefore, though S3 has, over time, gained features which oftne mimic a filesystem interface, in some cases it can behave very differently. In particular "empty directories" are, in actuality, 0-byte files and can have some unexpected behavior, e.g. there is no stat(dir) like in a true filesystem.

Min.io

Min.io is fully compatible with the S3 API and therefore this package can be used to interact with it. To use Min.io requires a dedicated AWS configuration object, see the Minio.jl package. This package also contains some convenience functions for easily setting up a server for experimentation and testing with the min.io/S3 interface.

+UInt8[]
Warning

S3 is a pure key-value store, NOT a filesystem. Therefore, though S3 has, over time, gained features which oftne mimic a filesystem interface, in some cases it can behave very differently. In particular "empty directories" are, in actuality, 0-byte files and can have some unexpected behavior, e.g. there is no stat(dir) like in a true filesystem.

Min.io

Min.io is fully compatible with the S3 API and therefore this package can be used to interact with it. To use Min.io requires a dedicated AWS configuration object, see the Minio.jl package. This package also contains some convenience functions for easily setting up a server for experimentation and testing with the min.io/S3 interface.

diff --git a/dev/search/index.html b/dev/search/index.html index 1fbeb564..5da1bdf3 100644 --- a/dev/search/index.html +++ b/dev/search/index.html @@ -1,2 +1,2 @@ -Search · AWSS3.jl

Loading search...

    +Search · AWSS3.jl

    Loading search...