diff --git a/dev/.documenter-siteinfo.json b/dev/.documenter-siteinfo.json index 001a4470f..c6ddf53ce 100644 --- a/dev/.documenter-siteinfo.json +++ b/dev/.documenter-siteinfo.json @@ -1 +1 @@ -{"documenter":{"julia_version":"1.9.4","generation_timestamp":"2023-11-20T08:58:41","documenter_version":"1.0.1"}} \ No newline at end of file +{"documenter":{"julia_version":"1.9.4","generation_timestamp":"2023-11-20T10:07:37","documenter_version":"1.0.1"}} \ No newline at end of file diff --git a/dev/getting-started/index.html b/dev/getting-started/index.html index 3521de275..bcf9d3876 100644 --- a/dev/getting-started/index.html +++ b/dev/getting-started/index.html @@ -63,4 +63,4 @@ startOnLoad: true, theme: "neutral" }); - + diff --git a/dev/guides/authentication/index.html b/dev/guides/authentication/index.html index 2da27513d..10438e150 100644 --- a/dev/guides/authentication/index.html +++ b/dev/guides/authentication/index.html @@ -8,4 +8,4 @@ startOnLoad: true, theme: "neutral" }); - + diff --git a/dev/guides/datasets/index.html b/dev/guides/datasets/index.html index 4bd0882d5..00e2a20f3 100644 --- a/dev/guides/datasets/index.html +++ b/dev/guides/datasets/index.html @@ -107,4 +107,4 @@ startOnLoad: true, theme: "neutral" }); - + diff --git a/dev/guides/jobs/index.html b/dev/guides/jobs/index.html index 441cdb357..7c0b865b4 100644 --- a/dev/guides/jobs/index.html +++ b/dev/guides/jobs/index.html @@ -117,4 +117,4 @@ startOnLoad: true, theme: "neutral" }); -
+
diff --git a/dev/index.html b/dev/index.html index c23e4de61..43451142c 100644 --- a/dev/index.html +++ b/dev/index.html @@ -4,4 +4,4 @@ startOnLoad: true, theme: "neutral" }); - + diff --git a/dev/internal/index.html b/dev/internal/index.html index 382e01b00..1d47b525e 100644 --- a/dev/internal/index.html +++ b/dev/internal/index.html @@ -1,7 +1,7 @@ -Internal · JuliaHub.jl
GitHub

Internal

_PackageBundler

JuliaHub._PackageBundler.bundle — Function
bundle(dir; output = "",  force=false, allownoenv=false, verbose = true) -> String

Creates a .tar file with the contents of dir as well as any packages that are either tracked by path (developed) outside dir or packages that are not tracked by the PkgServer. Artifacts are also bundled. The bundled packages and artifacts go into a .bundled/depot directory and is set up like a depot and can thus be made available by adding it to DEPOT_PATH.

.git and globs listed in .juliabundleignore are excluded form the bundle.

Returns the hex-encoded SHA256 of the Manifest.toml file that is packed into the appbundle. The return value is used when requesting a sysimage build, in which case we have to pass the manifest's hash with the submit request.

source
JuliaHub._PackageBundler.path_filterer — Function
path_filterer(top)

Returns a function that takes a file or directory path and checks whether that is excluded by the nearest .juliabundleignore file. The function will also ignore any .git files and directories.

The top argument specifies the highest directory up the tree that will be searched for the .juliabundleignore file.

The function will return false for any excluded files and true otherwise, and can be used as a predicate for filtering files that should be bundled.

source

Index

GitHub

Internal

_PackageBundler

JuliaHub._PackageBundler.bundle — Function
bundle(dir; output = "",  force=false, allownoenv=false, verbose = true) -> String

Creates a .tar file with the contents of dir as well as any packages that are either tracked by path (developed) outside dir or packages that are not tracked by the PkgServer. Artifacts are also bundled. The bundled packages and artifacts go into a .bundled/depot directory and is set up like a depot and can thus be made available by adding it to DEPOT_PATH.

.git and globs listed in .juliabundleignore are excluded form the bundle.

Returns the hex-encoded SHA256 of the Manifest.toml file that is packed into the appbundle. The return value is used when requesting a sysimage build, in which case we have to pass the manifest's hash with the submit request.

source
JuliaHub._PackageBundler.path_filterer — Function
path_filterer(top)

Returns a function that takes a file or directory path and checks whether that is excluded by the nearest .juliabundleignore file. The function will also ignore any .git files and directories.

The top argument specifies the highest directory up the tree that will be searched for the .juliabundleignore file.

The function will return false for any excluded files and true otherwise, and can be used as a predicate for filtering files that should be bundled.

source

Index

+
diff --git a/dev/reference/authentication/index.html b/dev/reference/authentication/index.html index 9ff2fa247..a9a86669f 100644 --- a/dev/reference/authentication/index.html +++ b/dev/reference/authentication/index.html @@ -1,13 +1,13 @@ -Authentication · JuliaHub.jl
GitHub

Authentication

In order to talk to a JuliaHub instance, you need to have a valid authentication token. JuliaHub reuses the Julia's built-in package server authentication tokens for this purpose. By default, the authentication uses the JULIA_PKG_SERVER environment variable to determine which JuliaHub instance to connect to, but this can be overridden by passing an argument to authenticate.

The authenticate function can be used to construct a token. If a valid token is available in ~/.julia/servers, it gets reused. Otherwise, a browser window is opened, starting an interactive authentication procedure.

All the functions that require authentication accept an auth keyword argument. However, JuliaHub.jl also stores the authentication token from the last authenticate call in a global variable and automatically uses that if auth is not provided, and also tries to authenticate automatically. The current global authentication object can be accessed via the current_authentication() function.

See also: authentication guide, authentication section on help.juliahub.com, PkgAuthentication.

Token expiration and refresh tokens

By default, JuliaHub access tokens expire in 24 hours. However, the tokens usually also have a refresh token, which is valid for 30 days. If the access token has expired, but there is a valid refresh token available, authenticate will automatically try to use that, to re-acquire an access token without starting an interactive authentication.

In JuliaHub job and cloud IDE environments, the authentication token on disk will be continuously kept up to date. The reauthenticate! function can be used to reload the token from disk.

Reference

JuliaHub.authenticate — Function
JuliaHub.authenticate(server = Pkg.pkg_server(); force::Bool = false, maxcount::Integer = 3, [hook::Base.Callable])

Authenticates with a JuliaHub server. If a valid authentication token does not exist in the Julia depot, a new token is acquired via an interactive browser based prompt. Returns an Authentication object if the authentication was successful, or throws an AuthenticationError if authentication fails.

The interactive prompts tries to authenticate for a maximum of maxcount times. If force is set to true, an existing authentication token is first deleted. This can be useful when the existing authentication token is causing the authentication to fail.

Extended help

By default, it attemps to connect to the currently configured Julia package server URL (configured e.g. via the JULIA_PKG_SERVER environment variable). However, this can be overridden by passing the server argument.

hook can be set to a function taking a single string-type argument, and will be passed the authorization URL the user should interact with in the browser. This can be used to override the default behavior coming from PkgAuthentication.

The returned Authentication object is also cached globally (overwriting any previously cached authentications), making it unnecessary to pass the returned object manually to other function calls. This is useful for interactive use, but should not be used in library code, as different authentication calls may clash.

source
JuliaHub.Authentication — Type
mutable struct Authentication

Authentication object constructed by the authenticate function that can be passed to the various JuliaHub.jl function via the auth keyword argument.

Objects have the following properties:

  • server :: URIs.URI: URL of the JuliaHub instance this authentication token applies to.
  • username :: String: user's JuliaHub username (used for e.g. to namespace datasets)
  • token :: JuliaHub.Secret: a Secret object storing the JuliaHub authentication token

Note that the object is mutable, and hence will be shared as it is passed around. And at the same time, functions such as reauthenticate! may modify the object.

See also: authenticate, reauthenticate!, current_authentication.

No public constructors

Objects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.

source
JuliaHub.current_authentication — Function
JuliaHub.current_authentication() -> Union{Authentication, Nothing}

Returns the current globally active Authentication object, or nothing if authenticate has not yet been called.

julia> JuliaHub.current_authentication()
-JuliaHub.Authentication("https://juliahub.com", "username", *****)
Note

Calling this function will not initialize authentication.

source
JuliaHub.check_authentication — Function
JuliaHub.check_authentication(; [auth::Authentication]) -> Bool

Checks if the authentication to a JuliaHub instance is still valid or not.

This can be used to periodically check an authentication token, to see if it is necessary to re-authenticate.

See also: reauthenticate!.

source
JuliaHub.reauthenticate! — Function
JuliaHub.reauthenticate!([auth::Authentication]; force::Bool = false, maxcount::Integer = 3, [hook::Base.Callable])

Attempts to update the authentication token in auth:

  • If the original auth.toml file has been updated, it simply reloads the token from the file.
  • If loading from auth.toml fails or force=true, it will attempt to re-authenticate with the server, possibly interactively.

If auth is omitted, it will reauthenticate the global Authentication object. The force, maxcount and hook are relevant for interactive authentication, and behave the same way as in the authenticate function.

This is mostly meant to be used to re-acquire authentication tokens in long-running sessions, where the initial authentication token may have expired.

As Authentication objects are mutable, the token will be updated in all contexts where the reference to the Authentication has been passed to.

See also: authenticate, current_authentication, Authentication, check_authentication.

source
JuliaHub.Secret — Type
mutable struct Secret

A helper type for storing secrets. Internally it is a covenience wrapper around Base.SecretBuffer. Predominantly used in Authentication objects to store the JuliaHub authentication token.

The String(::Secret) function can be used to obtain an unsecure string copy of the secret stored in the object.

julia> s = JuliaHub.Secret("secret-string")
+Authentication · JuliaHub.jl
GitHub
Authentication
In order to talk to a JuliaHub instance, you need to have a valid authentication token. JuliaHub reuses the Julia's built-in package server authentication tokens for this purpose. By default, the authentication uses the JULIA_PKG_SERVER environment variable to determine which JuliaHub instance to connect to, but this can be overridden by passing an argument to authenticate.
The authenticate function can be used to construct a token. If a valid token is available in ~/.julia/servers, it gets reused. Otherwise, a browser window is opened, starting an interactive authentication procedure.
All the functions that require authentication accept an auth keyword argument. However, JuliaHub.jl also stores the authentication token from the last authenticate call in a global variable and automatically uses that if auth is not provided, and also tries to authenticate automatically. The current global authentication object can be accessed via the current_authentication() function.
See also: authentication guide, authentication section on help.juliahub.com, PkgAuthentication.
Token expiration and refresh tokens
By default, JuliaHub access tokens expire in 24 hours. However, the tokens usually also have a refresh token, which is valid for 30 days. If the access token has expired, but there is a valid refresh token available, authenticate will automatically try to use that, to re-acquire an access token without starting an interactive authentication.
In JuliaHub job and cloud IDE environments, the authentication token on disk will be continuously kept up to date. The reauthenticate! function can be used to reload the token from disk.
Reference
JuliaHub.authenticate — Function
JuliaHub.authenticate(server = Pkg.pkg_server(); force::Bool = false, maxcount::Integer = 3, [hook::Base.Callable])
Authenticates with a JuliaHub server. If a valid authentication token does not exist in the Julia depot, a new token is acquired via an interactive browser based prompt. Returns an Authentication object if the authentication was successful, or throws an AuthenticationError if authentication fails.
The interactive prompts tries to authenticate for a maximum of maxcount times. If force is set to true, an existing authentication token is first deleted. This can be useful when the existing authentication token is causing the authentication to fail.
Extended help
By default, it attemps to connect to the currently configured Julia package server URL (configured e.g. via the JULIA_PKG_SERVER environment variable). However, this can be overridden by passing the server argument.
hook can be set to a function taking a single string-type argument, and will be passed the authorization URL the user should interact with in the browser. This can be used to override the default behavior coming from PkgAuthentication.
The returned Authentication object is also cached globally (overwriting any previously cached authentications), making it unnecessary to pass the returned object manually to other function calls. This is useful for interactive use, but should not be used in library code, as different authentication calls may clash.
source
JuliaHub.Authentication — Type
mutable struct Authentication
Authentication object constructed by the authenticate function that can be passed to the various JuliaHub.jl function via the auth keyword argument.
Objects have the following properties:
  • server :: URIs.URI: URL of the JuliaHub instance this authentication token applies to.
  • username :: String: user's JuliaHub username (used for e.g. to namespace datasets)
  • token :: JuliaHub.Secret: a Secret object storing the JuliaHub authentication token
Note that the object is mutable, and hence will be shared as it is passed around. And at the same time, functions such as reauthenticate! may modify the object.
See also: authenticate, reauthenticate!, current_authentication.
No public constructors
Objects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.
source
JuliaHub.current_authentication — Function
JuliaHub.current_authentication() -> Union{Authentication, Nothing}
Returns the current globally active Authentication object, or nothing if authenticate has not yet been called.
julia> JuliaHub.current_authentication()
+JuliaHub.Authentication("https://juliahub.com", "username", *****)
Note
Calling this function will not initialize authentication.
source
JuliaHub.check_authentication — Function
JuliaHub.check_authentication(; [auth::Authentication]) -> Bool
Checks if the authentication to a JuliaHub instance is still valid or not.
This can be used to periodically check an authentication token, to see if it is necessary to re-authenticate.
See also: reauthenticate!.
source
JuliaHub.reauthenticate! — Function
JuliaHub.reauthenticate!([auth::Authentication]; force::Bool = false, maxcount::Integer = 3, [hook::Base.Callable])
Attempts to update the authentication token in auth:
  • If the original auth.toml file has been updated, it simply reloads the token from the file.
  • If loading from auth.toml fails or force=true, it will attempt to re-authenticate with the server, possibly interactively.
If auth is omitted, it will reauthenticate the global Authentication object. The force, maxcount and hook are relevant for interactive authentication, and behave the same way as in the authenticate function.
This is mostly meant to be used to re-acquire authentication tokens in long-running sessions, where the initial authentication token may have expired.
As Authentication objects are mutable, the token will be updated in all contexts where the reference to the Authentication has been passed to.
See also: authenticate, current_authentication, Authentication, check_authentication.
source
JuliaHub.Secret — Type
mutable struct Secret
A helper type for storing secrets. Internally it is a covenience wrapper around Base.SecretBuffer. Predominantly used in Authentication objects to store the JuliaHub authentication token.
The String(::Secret) function can be used to obtain an unsecure string copy of the secret stored in the object.
julia> s = JuliaHub.Secret("secret-string")
 JuliaHub.Secret("*******")
 
 julia> String(s)
 "secret-string"
Constructors
Secret(::AbstractString)
-Secret(::Vector{UInt8})
Create a Secret object from the input strings.
source
Index

+
diff --git a/dev/reference/datasets/index.html b/dev/reference/datasets/index.html index 0f7cb7461..2e61e1f99 100644 --- a/dev/reference/datasets/index.html +++ b/dev/reference/datasets/index.html @@ -1,5 +1,5 @@ -Datasets · JuliaHub.jl
GitHub

Datasets

These APIs allow you to create, read, update, and delete datasets owned by the currently authenticated user.

See also: help.julialang.org on datasets, DataSets.jl.

Dataset types

JuliaHub currently has two distinct types of datasets:

  1. Blob: a single file; or, more abstractly, a collection of bytes
  2. BlobTree: a directory or a file; more abstractly a tree-like collection of Blobs, indexed by file system paths

These types mirror the concepts in DataSets.jl

JuliaHub.jl APIs do not rely that much on the dataset type for anything, except when downloading or uploading. In that case, a local file always corresponds to a Blob, and a local directory corresponds to a BlobTree. For example, when trying to upload a file as a new version of a BlobTree-type dataset will fail, because the dataset type can not change.

The upload_dataset function uses information filesystem to determine whether the created dataset is a Blob or a BlobTree, and similarly download_dataset will always download a Blob into a file, and a BlobTree as a directory.

Dataset versions

A JuliaHub dataset can have zero or more versions. A newly created dataset usually has at least one version, but it may have zero versions if, for example, the upload did not finish. The versions are indexed with a linear list of integers starting from 1.

Reference

JuliaHub.Dataset — Type
struct Dataset

Information about a dataset stored on JuliaHub, and the following fields are considered to be public API:

  • uuid :: UUID: dataset UUID
  • owner :: String: username of the dataset owner
  • name :: String: dataset name
  • dtype :: String: generally either Blob or BlobTree, but additional values may be added in the future
  • versions :: Vector{DatasetVersion}: an ordered list of DatasetVersion objects, one for each dataset version, sorted from oldest to latest (i.e. you can use last to get the newest version).
  • size :: Int: total size of the whole dataset (including all the dataset versions) in bytes
  • Fields to access user-provided dataset metadata:
    • description :: String: dataset description
    • tags :: Vector{String}: a list of tags
Canonical fully qualified dataset name

In some contexts, like when accessing JuliaHub datasets with DataSets.jl, the .owner-.name tuple constitutes the fully qualifed dataset name, uniquely identifying a dataset on a JuliaHub instance. I.e. for a dataset object dataset, it can be constructed as "$(dataset.owner)/$(dataset.name)".

Non-dynamic dataset objects

Dataset objects represents the dataset metadata when the Julia object was created (e.g. with dataset), and are not automatically kept up to date. To refresh the dataset metadata, you can pass the existing Dataset to JuliaHub.dataset.

No public constructors

Objects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.

source
JuliaHub.DatasetVersion — Type
struct DatasetVersion

Represents one version of a dataset.

Objects have the following properties:

  • .id: unique dataset version identifier (used e.g. in download_dataset to identify the dataset version).
  • .size :: Int: size of the dataset version in bytes
  • .timestamp :: ZonedDateTime: dataset version timestamp
julia> JuliaHub.datasets()

See also: Dataset, datasets, dataset.

No public constructors

Objects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.

source
JuliaHub.datasets — Function
JuliaHub.datasets([username::AbstractString]; shared::Bool=false, [auth::Authentication]) -> Vector{Dataset}

List all datasets owned by username, returning a list of Dataset objects.

If username is omitted, it returns the datasets owned by the currently authenticated user. If username is different from the currently authenticated user, it only returns the datasets that are readable to (i.e. somehow shared with) the currently authenticated user.

If shared = true, it also returns datasets that belong to other users that have that have been shared with the currently authenticated user. In this case, username is effectively ignored.

julia> JuliaHub.datasets()
+Datasets · JuliaHub.jl
GitHub
Datasets
These APIs allow you to create, read, update, and delete datasets owned by the currently authenticated user.
See also: help.julialang.org on datasets, DataSets.jl.
Dataset types
JuliaHub currently has two distinct types of datasets:
  1. Blob: a single file; or, more abstractly, a collection of bytes
  2. BlobTree: a directory or a file; more abstractly a tree-like collection of Blobs, indexed by file system paths
These types mirror the concepts in DataSets.jl
JuliaHub.jl APIs do not rely that much on the dataset type for anything, except when downloading or uploading. In that case, a local file always corresponds to a Blob, and a local directory corresponds to a BlobTree. For example, when trying to upload a file as a new version of a BlobTree-type dataset will fail, because the dataset type can not change.
The upload_dataset function uses information filesystem to determine whether the created dataset is a Blob or a BlobTree, and similarly download_dataset will always download a Blob into a file, and a BlobTree as a directory.
Dataset versions
A JuliaHub dataset can have zero or more versions. A newly created dataset usually has at least one version, but it may have zero versions if, for example, the upload did not finish. The versions are indexed with a linear list of integers starting from 1.
Reference
JuliaHub.Dataset — Type
struct Dataset
Information about a dataset stored on JuliaHub, and the following fields are considered to be public API:
  • uuid :: UUID: dataset UUID
  • owner :: String: username of the dataset owner
  • name :: String: dataset name
  • dtype :: String: generally either Blob or BlobTree, but additional values may be added in the future
  • versions :: Vector{DatasetVersion}: an ordered list of DatasetVersion objects, one for each dataset version, sorted from oldest to latest (i.e. you can use last to get the newest version).
  • size :: Int: total size of the whole dataset (including all the dataset versions) in bytes
  • Fields to access user-provided dataset metadata:
    • description :: String: dataset description
    • tags :: Vector{String}: a list of tags
Canonical fully qualified dataset name
In some contexts, like when accessing JuliaHub datasets with DataSets.jl, the .owner-.name tuple constitutes the fully qualifed dataset name, uniquely identifying a dataset on a JuliaHub instance. I.e. for a dataset object dataset, it can be constructed as "$(dataset.owner)/$(dataset.name)".
Non-dynamic dataset objects
Dataset objects represents the dataset metadata when the Julia object was created (e.g. with dataset), and are not automatically kept up to date. To refresh the dataset metadata, you can pass the existing Dataset to JuliaHub.dataset.
No public constructors
Objects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.
source
JuliaHub.DatasetVersion — Type
struct DatasetVersion
Represents one version of a dataset.
Objects have the following properties:
  • .id: unique dataset version identifier (used e.g. in download_dataset to identify the dataset version).
  • .size :: Int: size of the dataset version in bytes
  • .timestamp :: ZonedDateTime: dataset version timestamp
julia> JuliaHub.datasets()
See also: Dataset, datasets, dataset.
No public constructors
Objects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.
source
JuliaHub.datasets — Function
JuliaHub.datasets([username::AbstractString]; shared::Bool=false, [auth::Authentication]) -> Vector{Dataset}
List all datasets owned by username, returning a list of Dataset objects.
If username is omitted, it returns the datasets owned by the currently authenticated user. If username is different from the currently authenticated user, it only returns the datasets that are readable to (i.e. somehow shared with) the currently authenticated user.
If shared = true, it also returns datasets that belong to other users that have that have been shared with the currently authenticated user. In this case, username is effectively ignored.
julia> JuliaHub.datasets()
 2-element Vector{JuliaHub.Dataset}:
  JuliaHub.dataset(("username", "example-dataset"))
  JuliaHub.dataset(("username", "blobtree/example"))
@@ -12,7 +12,7 @@
 
 julia> JuliaHub.datasets("anotheruser")
 1-element Vector{JuliaHub.Dataset}:
- JuliaHub.dataset(("anotheruser", "publicdataset"))
Non-dynamic dataset objects
Dataset objects represents the dataset metadata when the Julia object was created (e.g. with dataset), and are not automatically kept up to date. To refresh the dataset metadata, you can pass the existing Dataset to JuliaHub.dataset.
source
JuliaHub.DatasetReference — Type
const DatasetReference :: Type
Type constraint on the first argument of most of the datasets-related functions, that is used to uniquely specify the dataset that the operation will affect.
There are three different objects that can be passed as a dataset reference (dsref::DatasetReference):
  • (owner::AbstractString, dataset_name::AbstractString)::Tuple{AbstractString,AbstractString}
    A tuple of the owner's username and the dataset's name.
  • dataset_name::AbstractString
    Just a string with the dataset name; in this case the dataset's owner will be assumed to be the currently authenticated user (with the username determined from the Authentication objects passed via the auth keyword).
  • dataset::Dataset
    Uses the owner and dataset name information from a Dataset object.
No UUID mismatch checks
When using the third option (i.e. passing a Dataset), the dataset UUID will not be checked. So if the dataset with the same owner and username has been deleted and re-created as a new dataset (potentially of a different dtype etc), the functions will then act on the new dataset.
source
JuliaHub.dataset — Function
JuliaHub.dataset(dataset::DatasetReference; throw::Bool=true, [auth::Authentication]) -> Dataset
Looks up a dataset based on the dataset reference dataset. Returns the Dataset object corresponding to dataset_name, or throws a InvalidRequestError if the dataset can not be found (if throw=false is passed, returns nothing instead).
By passing a Dataset object as dataset, this can be used to update the Dataset object.
julia> dataset = JuliaHub.dataset("example-dataset")
+ JuliaHub.dataset(("anotheruser", "publicdataset"))
Non-dynamic dataset objects
Dataset objects represents the dataset metadata when the Julia object was created (e.g. with dataset), and are not automatically kept up to date. To refresh the dataset metadata, you can pass the existing Dataset to JuliaHub.dataset.
source
JuliaHub.DatasetReference — Type
const DatasetReference :: Type
Type constraint on the first argument of most of the datasets-related functions, that is used to uniquely specify the dataset that the operation will affect.
There are three different objects that can be passed as a dataset reference (dsref::DatasetReference):
  • (owner::AbstractString, dataset_name::AbstractString)::Tuple{AbstractString,AbstractString}
    A tuple of the owner's username and the dataset's name.
  • dataset_name::AbstractString
    Just a string with the dataset name; in this case the dataset's owner will be assumed to be the currently authenticated user (with the username determined from the Authentication objects passed via the auth keyword).
  • dataset::Dataset
    Uses the owner and dataset name information from a Dataset object.
No UUID mismatch checks
When using the third option (i.e. passing a Dataset), the dataset UUID will not be checked. So if the dataset with the same owner and username has been deleted and re-created as a new dataset (potentially of a different dtype etc), the functions will then act on the new dataset.
source
JuliaHub.dataset — Function
JuliaHub.dataset(dataset::DatasetReference; throw::Bool=true, [auth::Authentication]) -> Dataset
Looks up a dataset based on the dataset reference dataset. Returns the Dataset object corresponding to dataset_name, or throws a InvalidRequestError if the dataset can not be found (if throw=false is passed, returns nothing instead).
By passing a Dataset object as dataset, this can be used to update the Dataset object.
julia> dataset = JuliaHub.dataset("example-dataset")
 Dataset: example-dataset (Blob)
  owner: username
  description: An example dataset
@@ -26,12 +26,12 @@
  description: An example dataset
  versions: 2
  size: 388 bytes
- tags: tag1, tag2
If the specifed username is not the currently authenticated user, the dataset must be shared with the currently authenticated user (i.e. contained in datasets(; shared=true)).
Note
This will call datasets every time, which might become a problem if you are processing a large number of datasets. In that case, you should call datasets and process the returned list yourself.
Non-dynamic dataset objects
Dataset objects represents the dataset metadata when the Julia object was created (e.g. with dataset), and are not automatically kept up to date. To refresh the dataset metadata, you can pass the existing Dataset to JuliaHub.dataset.
source
JuliaHub.download_dataset — Function
download_dataset(
+ tags: tag1, tag2
If the specifed username is not the currently authenticated user, the dataset must be shared with the currently authenticated user (i.e. contained in datasets(; shared=true)).
Note
This will call datasets every time, which might become a problem if you are processing a large number of datasets. In that case, you should call datasets and process the returned list yourself.
Non-dynamic dataset objects
Dataset objects represents the dataset metadata when the Julia object was created (e.g. with dataset), and are not automatically kept up to date. To refresh the dataset metadata, you can pass the existing Dataset to JuliaHub.dataset.
source
JuliaHub.download_dataset — Function
download_dataset(
     dataset::DatasetReference, local_path::AbstractString;
     replace::Bool = false, [version::Integer],
     [quiet::Bool = false], [auth::Authentication]
-) -> String
Downloads the dataset specified by the dataset reference dataset to local_path (which must not exist, unless replace = true), returning the absolute path to the downloaded file or directory. If the dataset is a Blob, then the created local_path will be a file, and if the dataset is a BlobTree the local_path will be a directory.
By default, it downloads the latest version, but an older version can be downloaded by specifying the version keyword argument. Caution: you should never assume that the index of the .versions property of Dataset matches the version number – always explicitly use the .id propert of the DatasetVersion object.
The function also prints download progress to standard output. This can be disabled by setting quiet=true. Any error output from the download is still printed.
Warning
Setting replace = true will recursively erase any existing data at local_path before replacing it with the dataset contents.
source
JuliaHub.upload_dataset — Function
JuliaHub.upload_dataset(dataset::DatasetReference, local_path; [auth,] kwargs...) -> Dataset
Uploads a new dataset or a new version of an existing dataset, with the dataset specified by the dataset reference dataset. The dataset type is determined from the local path (Blob if a file, BlobTree if a directory). If a Dataset object is passed, it attempts to update that dataset. Returns an updated Dataset object.
The following keyword arguments can be used to control the exact behavior of the function:
  • create :: Bool (default: true): Create the dataset, if it already does not exist.
  • update :: Bool (default: false): Upload the data as a new dataset version, if the dataset exists.
  • replace :: Bool (default: false): If a dataset exists, delete all existing data and create a new dataset with the same name instead. Excludes update = true, and only creates a completely new dataset if create=true as well.
In addition, the following keyword arguments can be passed to set or updated the dataset metadata when uploading:
  • description: description of the dataset (a string)
  • tags: an iterable of strings of all the tags of the dataset
  • visibility: a string with possible values public or private
  • license: a valid SPDX license identifier, or a tuple (:fulltext, license_text), where license_text is the full text string of a custom license
  • groups: an iterable of valid group names
If a dataset already exists, then these fields are updated as if update_dataset was called.
The function will throw an ArgumentError for invalid argument combinations.
Note
Presently, it is only possible to upload datasets for the currently authenticated user.
source
JuliaHub.update_dataset — Function
JuliaHub.update_dataset(dataset::DatasetReference; kwargs..., [auth]) -> Dataset
Updates the metadata of the dataset specified by the dataset reference dataset, as according to the keyword arguments keyword arguments. If the keywords are omitted, the metadata corresponding to it remains unchanged. Returns the Dataset object corresponding to the updated dataset.
The supported keywords are:
  • description: description of the dataset (a string)
  • tags: an iterable of strings of all the tags of the dataset
  • visibility: a string with possible values public or private
  • license: a valid SPDX license identifier, or a tuple (:fulltext, license_text), where license_text is the full text string of a custom license
  • groups: an iterable of valid group names
For example, to add a new tag to a dataset:
dataset = JuliaHub.dataset("my_dataset")
-JuliaHub.update(dataset; tags = [dataset.tags..., "newtag"])
Note
Presently, it is only possible to update datasets for the currently authenticated user.
source
JuliaHub.delete_dataset — Function
JuliaHub.delete_dataset(dataset::DatasetReference; force::Bool=false, [auth::Authentication]) -> Nothing
Delete the dataset specified by the dataset reference dataset. Will return nothing if the delete was successful, or throws an error if it was not.
Normally, when the dataset to be deleted does not exist, the function throws an error. This can be overridden by setting force = true.
julia> JuliaHub.datasets()
+) -> String
Downloads the dataset specified by the dataset reference dataset to local_path (which must not exist, unless replace = true), returning the absolute path to the downloaded file or directory. If the dataset is a Blob, then the created local_path will be a file, and if the dataset is a BlobTree the local_path will be a directory.
By default, it downloads the latest version, but an older version can be downloaded by specifying the version keyword argument. Caution: you should never assume that the index of the .versions property of Dataset matches the version number – always explicitly use the .id propert of the DatasetVersion object.
The function also prints download progress to standard output. This can be disabled by setting quiet=true. Any error output from the download is still printed.
Warning
Setting replace = true will recursively erase any existing data at local_path before replacing it with the dataset contents.
source
JuliaHub.upload_dataset — Function
JuliaHub.upload_dataset(dataset::DatasetReference, local_path; [auth,] kwargs...) -> Dataset
Uploads a new dataset or a new version of an existing dataset, with the dataset specified by the dataset reference dataset. The dataset type is determined from the local path (Blob if a file, BlobTree if a directory). If a Dataset object is passed, it attempts to update that dataset. Returns an updated Dataset object.
The following keyword arguments can be used to control the exact behavior of the function:
  • create :: Bool (default: true): Create the dataset, if it already does not exist.
  • update :: Bool (default: false): Upload the data as a new dataset version, if the dataset exists.
  • replace :: Bool (default: false): If a dataset exists, delete all existing data and create a new dataset with the same name instead. Excludes update = true, and only creates a completely new dataset if create=true as well.
In addition, the following keyword arguments can be passed to set or updated the dataset metadata when uploading:
  • description: description of the dataset (a string)
  • tags: an iterable of strings of all the tags of the dataset
  • visibility: a string with possible values public or private
  • license: a valid SPDX license identifier, or a tuple (:fulltext, license_text), where license_text is the full text string of a custom license
  • groups: an iterable of valid group names
If a dataset already exists, then these fields are updated as if update_dataset was called.
The function will throw an ArgumentError for invalid argument combinations.
Note
Presently, it is only possible to upload datasets for the currently authenticated user.
source
JuliaHub.update_dataset — Function
JuliaHub.update_dataset(dataset::DatasetReference; kwargs..., [auth]) -> Dataset
Updates the metadata of the dataset specified by the dataset reference dataset, as according to the keyword arguments keyword arguments. If the keywords are omitted, the metadata corresponding to it remains unchanged. Returns the Dataset object corresponding to the updated dataset.
The supported keywords are:
  • description: description of the dataset (a string)
  • tags: an iterable of strings of all the tags of the dataset
  • visibility: a string with possible values public or private
  • license: a valid SPDX license identifier, or a tuple (:fulltext, license_text), where license_text is the full text string of a custom license
  • groups: an iterable of valid group names
For example, to add a new tag to a dataset:
dataset = JuliaHub.dataset("my_dataset")
+JuliaHub.update(dataset; tags = [dataset.tags..., "newtag"])
Note
Presently, it is only possible to update datasets for the currently authenticated user.
source
JuliaHub.delete_dataset — Function
JuliaHub.delete_dataset(dataset::DatasetReference; force::Bool=false, [auth::Authentication]) -> Nothing
Delete the dataset specified by the dataset reference dataset. Will return nothing if the delete was successful, or throws an error if it was not.
Normally, when the dataset to be deleted does not exist, the function throws an error. This can be overridden by setting force = true.
julia> JuliaHub.datasets()
 2-element Vector{JuliaHub.Dataset}:
  JuliaHub.dataset(("username", "example-dataset"))
  JuliaHub.dataset(("username", "blobtree"))
@@ -40,9 +40,9 @@
 
 julia> JuliaHub.datasets()
 1-element Vector{JuliaHub.Dataset}:
- JuliaHub.dataset(("username", "blobtree"))
Note
Presently, it is only possible to delete datasets for the currently authenticated user.
source
Index

+
diff --git a/dev/reference/exceptions/index.html b/dev/reference/exceptions/index.html index 5a759df07..c9bc4fc9e 100644 --- a/dev/reference/exceptions/index.html +++ b/dev/reference/exceptions/index.html @@ -1,7 +1,7 @@ -Exceptions · JuliaHub.jl
GitHub

Exceptions

JuliaHub.jl is designed in a way that the only errors it should throw under normal circumstances are subtypes of JuliaHubException (in addition to standard ArgumentErrors and MethodErrors etc. from invalid function calls). Any unhandled errors from JuliaHub.jl or its dependencies should be considered a bug.

Debugging JuliaHub.jl issues

You can also enable debug logging for JuliaHub, which will make JuliaHub.jl print out additional debug messages, by setting the JULIA_DEBUG environment variable:

ENV["JULIA_DEBUG"]="JuliaHub"
JuliaHub.AuthenticationError — Type
struct AuthenticationError <: JuliaHubException

Exception thrown if the authentication fails. The .msg fields contains a human-readable error message.

source
JuliaHub.InvalidRequestError — Type
struct InvalidRequestError <: JuliaHubException

An exception thrown if the request was rejected by the backend due to request parameters that are inconsistent with the backend state. The .msg field contains the error message.

source
JuliaHub.JuliaHubConnectionError — Type
struct JuliaHubConnectionError <: JuliaHubException

An exception thrown if there is a communication error with JuliaHub.

The .msg field contains the error message. If there is an underlying exception, it is stored in the .exception field.

source
JuliaHub.JuliaHubError — Type
struct JuliaHubError <: JuliaHubException

An exception thrown if there is an unexpected response from or backend failure in JuliaHub.

The .msg field contains the error message. If there is an underlying exception, it is stored in the .exception field.

source
JuliaHub.PermissionError — Type
struct PermissionError <: JuliaHubException

Thrown if the currently authenticated user does not have the necessary permissions to perform the operation. The .msg field contains the error message, and .response may contain the raw server response.

source

Index

GitHub

Exceptions

JuliaHub.jl is designed in a way that the only errors it should throw under normal circumstances are subtypes of JuliaHubException (in addition to standard ArgumentErrors and MethodErrors etc. from invalid function calls). Any unhandled errors from JuliaHub.jl or its dependencies should be considered a bug.

Debugging JuliaHub.jl issues

You can also enable debug logging for JuliaHub, which will make JuliaHub.jl print out additional debug messages, by setting the JULIA_DEBUG environment variable:

ENV["JULIA_DEBUG"]="JuliaHub"
JuliaHub.AuthenticationError — Type
struct AuthenticationError <: JuliaHubException

Exception thrown if the authentication fails. The .msg fields contains a human-readable error message.

source
JuliaHub.InvalidRequestError — Type
struct InvalidRequestError <: JuliaHubException

An exception thrown if the request was rejected by the backend due to request parameters that are inconsistent with the backend state. The .msg field contains the error message.

source
JuliaHub.JuliaHubConnectionError — Type
struct JuliaHubConnectionError <: JuliaHubException

An exception thrown if there is a communication error with JuliaHub.

The .msg field contains the error message. If there is an underlying exception, it is stored in the .exception field.

source
JuliaHub.JuliaHubError — Type
struct JuliaHubError <: JuliaHubException

An exception thrown if there is an unexpected response from or backend failure in JuliaHub.

The .msg field contains the error message. If there is an underlying exception, it is stored in the .exception field.

source
JuliaHub.PermissionError — Type
struct PermissionError <: JuliaHubException

Thrown if the currently authenticated user does not have the necessary permissions to perform the operation. The .msg field contains the error message, and .response may contain the raw server response.

source

Index

+
diff --git a/dev/reference/job-submission/index.html b/dev/reference/job-submission/index.html index a9ae1b8af..e2cd2e390 100644 --- a/dev/reference/job-submission/index.html +++ b/dev/reference/job-submission/index.html @@ -86,27 +86,27 @@ - GPU: no - vCores: 2 - Memory: 8 Gb - - Price: 0.17 $/hr

They can be used to contruct explicit compute configuration objects when submitting JuliaHub jobs.

See also: submit_job, ComputeConfig.

source
JuliaHub.nodespecs — Function
JuliaHub.nodespecs(; auth::Authentication) -> Vector{NodeSpec}

Query node specifications available on the current server, returning a list of NodeSpec objects.

source
JuliaHub.nodespec — Function
JuliaHub.nodespec(
+ - Price: 0.17 $/hr

They can be used to contruct explicit compute configuration objects when submitting JuliaHub jobs.

See also: submit_job, ComputeConfig.

source
JuliaHub.nodespecs — Function
JuliaHub.nodespecs(; auth::Authentication) -> Vector{NodeSpec}

Query node specifications available on the current server, returning a list of NodeSpec objects.

source
JuliaHub.nodespec — Function
JuliaHub.nodespec(
     [nodes::Vector{NodeSpec}];
     ncpu::Integer=1, ngpu::Integer=false, memory::Integer=1,
     exactmatch::Bool=false, throw::Bool=true,
     [auth::Authentication]
-) -> Union{NodeSpec, Nothing}

Finds the node matching the specified node parameters. Throws an InvalidRequestError if it is unable to find a node with the specific parameters. However, if throw is set to false, it will return nothing instead in that situation.

By default, it searches for the smallest node that has the specified parameters or more higher. If exactmatch is set to true, it only returns a node specification if it can find one that matches the parameters exactly.

A list of nodes (e.g. from nodespecs) can also be passed, so that the function does not have to query the server for the list. When this method is used, it is not necessary to pass auth.

source
JuliaHub.BatchImage — Type
struct BatchImage

Represents an available JuliaHub batch job image. These can be passed to BatchJob to specify which underlying job image will be used for the job.

A list of available batch images can be accessed with batchimages and specific images can be constructed with batchimage.

No public constructors

Objects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.

See also: batchimages, batchimage, BatchJob, script, appbundle.

source
JuliaHub.batchimages — Function
JuliaHub.batchimages([product::AbstractString]; [auth::Authentication]) -> Vector{BatchImage}

Return the list of all batch job images available to the currently authenticated user, as a list of BatchImage objects. These can be passed to BatchJob.

Optionally, by passing a product identifier, the list can be narrowed down to images available for that specific product.

Batch images on older instances

When using the package with an older JuliaHub instance (<= 6.1), the non-default batch images show up with @legacy as the product name. This indicates that the package is using an older API, and not that the images themselves are outdated.

See also: BatchImage, batchimage, BatchJob, script, appbundle.

source
JuliaHub.batchimage — Function
JuliaHub.batchimage(
+) -> Union{NodeSpec, Nothing}

Finds the node matching the specified node parameters. Throws an InvalidRequestError if it is unable to find a node with the specific parameters. However, if throw is set to false, it will return nothing instead in that situation.

By default, it searches for the smallest node that has the specified parameters or more higher. If exactmatch is set to true, it only returns a node specification if it can find one that matches the parameters exactly.

A list of nodes (e.g. from nodespecs) can also be passed, so that the function does not have to query the server for the list. When this method is used, it is not necessary to pass auth.

source
JuliaHub.BatchImage — Type
struct BatchImage

Represents an available JuliaHub batch job image. These can be passed to BatchJob to specify which underlying job image will be used for the job.

A list of available batch images can be accessed with batchimages and specific images can be constructed with batchimage.

No public constructors

Objects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.

See also: batchimages, batchimage, BatchJob, script, appbundle.

source
JuliaHub.batchimages — Function
JuliaHub.batchimages([product::AbstractString]; [auth::Authentication]) -> Vector{BatchImage}

Return the list of all batch job images available to the currently authenticated user, as a list of BatchImage objects. These can be passed to BatchJob.

Optionally, by passing a product identifier, the list can be narrowed down to images available for that specific product.

Batch images on older instances

When using the package with an older JuliaHub instance (<= 6.1), the non-default batch images show up with @legacy as the product name. This indicates that the package is using an older API, and not that the images themselves are outdated.

See also: BatchImage, batchimage, BatchJob, script, appbundle.

source
JuliaHub.batchimage — Function
JuliaHub.batchimage(
     [product::AbstractString, [image::AbstractString]];
     throw::Bool=true, [auth::Authentication]
-) -> BatchImage

Pick a product job batch image from the list of all batch image, returning a BatchImage object. If image is omitted, it will return the default image corresponding to product. If product is omitted as well, it will return the default image of the instance (generally the standard Julia batch image).

Will throw an InvalidRequestError if the specified image can not be found. If throw=false, it will return nothing instead in this situation.

Batch images on older instances

When using the package with an older JuliaHub instance (<= 6.1), the non-default batch images show up with @legacy as the product name. This indicates that the package is using an older API, and not that the images themselves are outdated.

See also: BatchImage, batchimages, BatchJob, script, appbundle.

source
JuliaHub.AbstractJobConfig — Type
abstract type AbstractJobConfig

Abstract supertype of all application configuration types that can be passed to submit_job for submission as a JuliaHub job. The package has built-in support for the following application configurations:

source
JuliaHub.BatchJob — Type
struct BatchJob <: AbstractJobConfig

Represents the application configuration of a JuliaHub batch job. A batch job is defined by the following information:

  • The Julia code that is to be executed in the job.
  • Julia package environment (i.e. Project.toml, Manifest.toml) and other files, such as the appbundle.
  • The underlying batch job container image (see also batchimages), which defaults to the standard Julia image by default.

Instances of this types should normally not be constructed directly, and the following functions should be used instead:

  • script or @script_str: for submitting simple Julia scripts or code snippets
  • appbundle: for submitting more complex "appbundles" that include additional file, private or modified package dependencies etc.

Optional arguments

  • image :: Union{BatchImage, Nothing}: can be used to specify which product's batch job image will be used when running the job, by passing the appropriate BatchImage object (see also: batchimage and batchimages). If set to nothing (the default), the job runs with the default Julia image.

  • sysimage :: Bool: if set to true, requests that a system image is built from the job's Manifest.toml file before starting the job. Defaults to false.

JuliaHub compatibility

The sysimage = true option requires JuliaHub 6.3 to have an effect. When running against older JuliaHub versions, it does not have an effect.

Constructors

BatchJob(::BatchJob; [image::BatchImage], [sysimage::Bool]) -> BatchJob

Construct a BatchJob, but override some of the optional arguments documented above. When the argument is omitted, the value from the underlying BatchJob object is used. This is the only constructor that is part of the public API.

This method is particularly useful when used in in combination with the @script_str string macro, to be able to specify the job image or trigger a sysimage build. For example, the following snippet will set a different batch image for the script-type job:

JuliaHub.BatchJob(
+) -> BatchImage

Pick a product job batch image from the list of all batch image, returning a BatchImage object. If image is omitted, it will return the default image corresponding to product. If product is omitted as well, it will return the default image of the instance (generally the standard Julia batch image).

Will throw an InvalidRequestError if the specified image can not be found. If throw=false, it will return nothing instead in this situation.

Batch images on older instances

When using the package with an older JuliaHub instance (<= 6.1), the non-default batch images show up with @legacy as the product name. This indicates that the package is using an older API, and not that the images themselves are outdated.

See also: BatchImage, batchimages, BatchJob, script, appbundle.

source
JuliaHub.AbstractJobConfig — Type
abstract type AbstractJobConfig

Abstract supertype of all application configuration types that can be passed to submit_job for submission as a JuliaHub job. The package has built-in support for the following application configurations:

source
JuliaHub.BatchJob — Type
struct BatchJob <: AbstractJobConfig

Represents the application configuration of a JuliaHub batch job. A batch job is defined by the following information:

  • The Julia code that is to be executed in the job.
  • Julia package environment (i.e. Project.toml, Manifest.toml) and other files, such as the appbundle.
  • The underlying batch job container image (see also batchimages), which defaults to the standard Julia image by default.

Instances of this types should normally not be constructed directly, and the following functions should be used instead:

  • script or @script_str: for submitting simple Julia scripts or code snippets
  • appbundle: for submitting more complex "appbundles" that include additional file, private or modified package dependencies etc.

Optional arguments

  • image :: Union{BatchImage, Nothing}: can be used to specify which product's batch job image will be used when running the job, by passing the appropriate BatchImage object (see also: batchimage and batchimages). If set to nothing (the default), the job runs with the default Julia image.

  • sysimage :: Bool: if set to true, requests that a system image is built from the job's Manifest.toml file before starting the job. Defaults to false.

JuliaHub compatibility

The sysimage = true option requires JuliaHub 6.3 to have an effect. When running against older JuliaHub versions, it does not have an effect.

Constructors

BatchJob(::BatchJob; [image::BatchImage], [sysimage::Bool]) -> BatchJob

Construct a BatchJob, but override some of the optional arguments documented above. When the argument is omitted, the value from the underlying BatchJob object is used. This is the only constructor that is part of the public API.

This method is particularly useful when used in in combination with the @script_str string macro, to be able to specify the job image or trigger a sysimage build. For example, the following snippet will set a different batch image for the script-type job:

JuliaHub.BatchJob(
     JuliaHub.script"""
     @info "Hello World"
     """,
     image = JuliaHub.batchimage("...")
-)
source
JuliaHub.script — Function
JuliaHub.script(...) -> BatchJob

Constructs the configuration for a script-type batch job, returning the respective BatchJob object that can then be passed to submit_job. A script-type batch job is defined by the following:

  • A user-provided Julia script that gets executed on the server. Note that no validation of the input code is done.

  • An optional Julia package environment (e.g. Project.toml, Manifest.toml and Artifacts.toml). If any of the TOML files are provided, they must parse as valid TOML files, but no further validation is done client-side.

    If the manifest is not provided, the project environment must be instantiated from scratch, generally pulling in the latest versions of all the dependencies (although [compat] sections are honored).

    It is also fine to omit the project file, and just provide the manifest, and the environment defined by the manifest still gets instantiated. If both are omitted, the job runs in an empty environment.

  • A JuliaHub job image, which determines the container environment that will be used to execute the code in (see batchimage, batchimages, BatchImage). If omitted, the default Julia image is used.

See also the @script_str string macro to more easily submit simple scripts that are defined in code.

Methods

script(
+)
source
JuliaHub.script — Function
JuliaHub.script(...) -> BatchJob

Constructs the configuration for a script-type batch job, returning the respective BatchJob object that can then be passed to submit_job. A script-type batch job is defined by the following:

  • A user-provided Julia script that gets executed on the server. Note that no validation of the input code is done.

  • An optional Julia package environment (e.g. Project.toml, Manifest.toml and Artifacts.toml). If any of the TOML files are provided, they must parse as valid TOML files, but no further validation is done client-side.

    If the manifest is not provided, the project environment must be instantiated from scratch, generally pulling in the latest versions of all the dependencies (although [compat] sections are honored).

    It is also fine to omit the project file, and just provide the manifest, and the environment defined by the manifest still gets instantiated. If both are omitted, the job runs in an empty environment.

  • A JuliaHub job image, which determines the container environment that will be used to execute the code in (see batchimage, batchimages, BatchImage). If omitted, the default Julia image is used.

See also the @script_str string macro to more easily submit simple scripts that are defined in code.

Methods

script(
     scriptfile::AbstractString;
     [project_directory::AbstractString], [image::BatchImage], [sysimage::Bool]
 ) -> BatchJob

Constructs a script-type batch job configuration the will execute the code in scriptfile. Optionally, a path to a project environment directory can be passed via project_directory, which will be searched for the environment TOML files, and a job image can be specified via image.

script(;
     code::AbstractString,
     [project::AbstractString], [manifest::AbstractString], [artifacts::AbstractString],
     [image::BatchImage], [sysimage::Bool]
-) -> BatchJob

A lower-level method that can be used to construct the script-type BatchJob configuration directly in memory (i.e. without having to write out intermediate files).

The code keyword argument is mandatory and will specify contents of the Julia script that gets executed on the server. The Julia project environment can be specified by passing the contents of the TOML files via the corresponding arguments (project, manifest, artifacts). The job image can be specified via image.

Optional arguments

See BatchJob for a more thorough description of the optional arguments.

source
JuliaHub.@script_str — Macro
JuliaHub.@script_str -> JuliaHub.BatchJob

A string macro to conveniently construct a script-type batch job configuration (BatchJob) that can be submitted as a JuliaHub job.

script = JuliaHub.script"""
+) -> BatchJob

A lower-level method that can be used to construct the script-type BatchJob configuration directly in memory (i.e. without having to write out intermediate files).

The code keyword argument is mandatory and will specify contents of the Julia script that gets executed on the server. The Julia project environment can be specified by passing the contents of the TOML files via the corresponding arguments (project, manifest, artifacts). The job image can be specified via image.

Optional arguments

See BatchJob for a more thorough description of the optional arguments.

source
JuliaHub.@script_str — Macro
JuliaHub.@script_str -> JuliaHub.BatchJob

A string macro to conveniently construct a script-type batch job configuration (BatchJob) that can be submitted as a JuliaHub job.

script = JuliaHub.script"""
 @info "Hello World!"
 """

This allows for an easy submission of simple single-script jobs to JuliaHub:

JuliaHub.submit_job(
     JuliaHub.script"""
@@ -121,7 +121,7 @@
         """,
         image = JuliaHub.batchimage(...)
     )
-)

You can also use this pattern to set the sysimage option.

source
JuliaHub.appbundle — Function
JuliaHub.appbundle(
+)

You can also use this pattern to set the sysimage option.

source
JuliaHub.appbundle — Function
JuliaHub.appbundle(
     directory::AbstractString, codefile::AbstractString;
     [image::BatchImage], [sysimage::Bool]
 ) -> BatchJob
@@ -133,12 +133,12 @@
     code = """
     @show ENV
     """
-)

See BatchJob for a description of the optional arguments.

Extended help

The following should be kept in mind about how appbundles are handled:

  • The bundler looks for a Julia environment (i.e. Project.toml and/or Manifest.toml files) at the root of the directory. If the environment does not exist (i.e. the files are missing), one is created.

  • Development dependencies of the environment (i.e. packages added with pkg> develop or Pkg.develop()) are also bundled up into the archive that gets submitted to JuliaHub (including any current, uncommitted changes). Registered packages are installed via the package manager via the standard environment instantiation, and their source code is not included in the bundle directly.

  • When the JuliaHub job starts, the bundle is unpacked and the job's starting working directory is set to the appbundle/ directory, and you can e.g. load the data from those files with just read("my-data.txt", String).

    Note that @__DIR__ points elsewhere and, relatedly, include in the main script should be used with an absolute path (e.g. include(joinpath(pwd(), "my-julia-file.jl"))).

    JuliaHub 6.2 and older

    On some older JuliaHub versions (6.2 and older), the working directory was set to the parent directory of appbundle/, and so it was necessary to do joinpath("appbundle", "mydata.dat") to load the code.

source
JuliaHub.ComputeConfig — Type
struct ComputeConfig

This type encapsulates the configuration of a jobs's compute cluster, including the hardware configuration and the cluster topology.

See also: submit_job.

Constructors

JuliaHub.ComputeConfig(
+)

See BatchJob for a description of the optional arguments.

Extended help

The following should be kept in mind about how appbundles are handled:

  • The bundler looks for a Julia environment (i.e. Project.toml and/or Manifest.toml files) at the root of the directory. If the environment does not exist (i.e. the files are missing), one is created.

  • Development dependencies of the environment (i.e. packages added with pkg> develop or Pkg.develop()) are also bundled up into the archive that gets submitted to JuliaHub (including any current, uncommitted changes). Registered packages are installed via the package manager via the standard environment instantiation, and their source code is not included in the bundle directly.

  • When the JuliaHub job starts, the bundle is unpacked and the job's starting working directory is set to the appbundle/ directory, and you can e.g. load the data from those files with just read("my-data.txt", String).

    Note that @__DIR__ points elsewhere and, relatedly, include in the main script should be used with an absolute path (e.g. include(joinpath(pwd(), "my-julia-file.jl"))).

    JuliaHub 6.2 and older

    On some older JuliaHub versions (6.2 and older), the working directory was set to the parent directory of appbundle/, and so it was necessary to do joinpath("appbundle", "mydata.dat") to load the code.

source
JuliaHub.ComputeConfig — Type
struct ComputeConfig

This type encapsulates the configuration of a jobs's compute cluster, including the hardware configuration and the cluster topology.

See also: submit_job.

Constructors

JuliaHub.ComputeConfig(
     node::NodeSpec;
     nnodes::Integer = 1,
     process_per_node::Bool = true,
     elastic::Bool = false,
-)

  • node: a NodeSpec object that specifies the hardware of a single node.

  • nnodes::Union{Integer, Tuple{Integer, Integer}} = 1: specifies the number of nodes of type node that will be allocated. Alternatively, a two-integer tuple can also be passed, where the first value specifies the minimum number of nodes required to start a job. By default, a single-node job is started.

  • process_per_node::Bool = true: if true, there will only be a single Julia process per node, and the total number of Julia processes will be nnodes. If set to false, however, each core on each node will be allocated a separate Julia process (running in an isolated container on the same node), and so the total number of Julia processes will be nnodes × ncpu, and it will essentially always be a multi-process job.

  • elastic::Bool = false: if set, the job will be started in an elastic cluster mode. In this case, a minimum number of nnodes must not be passed.

source
JuliaHub.submit_job — Function
JuliaHub.submit_job(
+)

  • node: a NodeSpec object that specifies the hardware of a single node.

  • nnodes::Union{Integer, Tuple{Integer, Integer}} = 1: specifies the number of nodes of type node that will be allocated. Alternatively, a two-integer tuple can also be passed, where the first value specifies the minimum number of nodes required to start a job. By default, a single-node job is started.

  • process_per_node::Bool = true: if true, there will only be a single Julia process per node, and the total number of Julia processes will be nnodes. If set to false, however, each core on each node will be allocated a separate Julia process (running in an isolated container on the same node), and so the total number of Julia processes will be nnodes × ncpu, and it will essentially always be a multi-process job.

  • elastic::Bool = false: if set, the job will be started in an elastic cluster mode. In this case, a minimum number of nnodes must not be passed.

source
JuliaHub.submit_job — Function
JuliaHub.submit_job(
     app::Union{AbstractJuliaHubApp, AbstractJobConfig},
     [compute::ComputeConfig];
     # Compute keyword arguments
@@ -152,7 +152,7 @@
     # General keyword arguments
     dryrun::Bool = false,
     [auth :: Authentication]
-) -> Job

Submits the specified application config app as a job to JuliaHub. Returns a Job object corresponding to the submitted job.

Compute arguments. If compute is passed, the compute keyword arguments can not be passed. If compute is not passed, the following arguments can be used to specify the compute configration via keyword arguments:

  • ncpu, ngpu and memory are used to pick a node type that will be used to run the job. The node type will be a minimum one that satisfies the constraints, but may have more compute resources than specified by the arguments (it corresponds to the exactmatch = false case of nodespec).

  • nnodes, minimum_nnodes, process_per_node, and elastic specify the corresponding arguments in ComputeConfig.

Runtime configuration. These are used to set the Runtime configuration of the job.

  • alias :: Union{AbstractString, Nothing}: can be used to override the name of the job that gets displayed in the UI. Passing nothing is equivalent to omitting the argument.

  • timelimit :: Limit: sets the job's time limit (see Limit for valid values)

  • env: an iterable of key-value pairs that can be used to set environment variable that get set before, the job code gets executed.

  • project :: Union{UUID, AbstractString, Nothing}: the UUID of the project that the job will be associated with. If a string is passed, it must parse as a valid UUID. Passing nothing is equivalent to omitting the argument.

General arguments.

  • auth :: Authentication: optional authentication object (see the authentication section for more information)

  • dryrun :: Bool: if set to true, submit_job does not actually submit the job, but instead returns a WorkloadConfig object, which can be used to inspect the configuration that would be submitted.

    The WorkloadConfig object can then be submitted to JuliaHub with the additional submit_job method:

    JuliaHub.submit_job(::WorkloadConfig; [auth::Authentication])
JuliaHub compatibility

The timelimit = JuliaHub.Unlimited() argument requires JuliaHub 6.3+.

source
JuliaHub.Limit — Type
JuliaHub.Limit

Type-constraint on JuliaHub job timelimit arguments in submit_job.

The job time limit can either be a time period (an instance of Dates.Period), an Integer, (interpreted as the number of hours), or JuliaHub.Unlimited().

Only an integer number of hours are accepted by JuliaHub, and fractional hours from get rounded up to the next full integer number of hours (e.g. Dates.Minute(90) will be interpreted as 2 hours).

source
JuliaHub.Unlimited — Type
struct Unlimited

An instance of this type can be passed as the [timelimit] option to submit_job to start jobs that run indefinitely, until killed manually.

JuliaHub.submit_job(..., timelimit = JuliaHub.Unlimited())
source
JuliaHub.WorkloadConfig — Type
struct WorkloadConfig

Represents a full job configuration, including the application, compute and runtime configuration.

Instances of this type can be constructed by passing dryrun = true to submit_job, and can also be directly submitted to JuliaHub with the same function.

source

Experimental APIs

Experimental features

Starting application jobs with JuliaHub.jl is considered to be experimental. The APIs are likely to change in future JuliaHub.jl version.

JuliaHub.AbstractJuliaHubApp — Type
abstract type AbstractJuliaHubApp

Abstract supertype for JuliaHub applications object types.

Experimental API

Applications-related APIs are experimental, and may be changed or removed without notice.

source
JuliaHub.applications — Function
JuliaHub.applications([category::Symbol]; [auth::Authentication]) -> Vector{AbstractJuliaHubApp}

Returns the list of applications enabled for the authenticated user, optionally in the specified category only. Returns a vector of AbstractJuliaHubApp instances.

julia> JuliaHub.applications()
+) -> Job

Submits the specified application config app as a job to JuliaHub. Returns a Job object corresponding to the submitted job.

Compute arguments. If compute is passed, the compute keyword arguments can not be passed. If compute is not passed, the following arguments can be used to specify the compute configration via keyword arguments:

  • ncpu, ngpu and memory are used to pick a node type that will be used to run the job. The node type will be a minimum one that satisfies the constraints, but may have more compute resources than specified by the arguments (it corresponds to the exactmatch = false case of nodespec).

  • nnodes, minimum_nnodes, process_per_node, and elastic specify the corresponding arguments in ComputeConfig.

Runtime configuration. These are used to set the Runtime configuration of the job.

  • alias :: Union{AbstractString, Nothing}: can be used to override the name of the job that gets displayed in the UI. Passing nothing is equivalent to omitting the argument.

  • timelimit :: Limit: sets the job's time limit (see Limit for valid values)

  • env: an iterable of key-value pairs that can be used to set environment variable that get set before, the job code gets executed.

  • project :: Union{UUID, AbstractString, Nothing}: the UUID of the project that the job will be associated with. If a string is passed, it must parse as a valid UUID. Passing nothing is equivalent to omitting the argument.

General arguments.

  • auth :: Authentication: optional authentication object (see the authentication section for more information)

  • dryrun :: Bool: if set to true, submit_job does not actually submit the job, but instead returns a WorkloadConfig object, which can be used to inspect the configuration that would be submitted.

    The WorkloadConfig object can then be submitted to JuliaHub with the additional submit_job method:

    JuliaHub.submit_job(::WorkloadConfig; [auth::Authentication])
JuliaHub compatibility

The timelimit = JuliaHub.Unlimited() argument requires JuliaHub 6.3+.

source
JuliaHub.Limit — Type
JuliaHub.Limit

Type-constraint on JuliaHub job timelimit arguments in submit_job.

The job time limit can either be a time period (an instance of Dates.Period), an Integer, (interpreted as the number of hours), or JuliaHub.Unlimited().

Only an integer number of hours are accepted by JuliaHub, and fractional hours from get rounded up to the next full integer number of hours (e.g. Dates.Minute(90) will be interpreted as 2 hours).

source
JuliaHub.Unlimited — Type
struct Unlimited

An instance of this type can be passed as the [timelimit] option to submit_job to start jobs that run indefinitely, until killed manually.

JuliaHub.submit_job(..., timelimit = JuliaHub.Unlimited())
source
JuliaHub.WorkloadConfig — Type
struct WorkloadConfig

Represents a full job configuration, including the application, compute and runtime configuration.

Instances of this type can be constructed by passing dryrun = true to submit_job, and can also be directly submitted to JuliaHub with the same function.

source

Experimental APIs

Experimental features

Starting application jobs with JuliaHub.jl is considered to be experimental. The APIs are likely to change in future JuliaHub.jl version.

JuliaHub.AbstractJuliaHubApp — Type
abstract type AbstractJuliaHubApp

Abstract supertype for JuliaHub applications object types.

Experimental API

Applications-related APIs are experimental, and may be changed or removed without notice.

source
JuliaHub.applications — Function
JuliaHub.applications([category::Symbol]; [auth::Authentication]) -> Vector{AbstractJuliaHubApp}

Returns the list of applications enabled for the authenticated user, optionally in the specified category only. Returns a vector of AbstractJuliaHubApp instances.

julia> JuliaHub.applications()
 7-element Vector{JuliaHub.AbstractJuliaHubApp}:
  JuliaHub.application(:default, "Linux Desktop")
  JuliaHub.application(:default, "Julia IDE")
@@ -177,7 +177,7 @@
 julia> JuliaHub.applications(:user)
 1-element Vector{JuliaHub.UserApp}:
  JuliaHub.application(:user, "ExampleApp.jl")
-
Experimental API

Applications-related APIs are experimental, and may be changed or removed without notice.

source
JuliaHub.application — Function
JuliaHub.application(
+
Experimental API

Applications-related APIs are experimental, and may be changed or removed without notice.

source
JuliaHub.application — Function
JuliaHub.application(
     category::Symbol, name::AbstractString;
     throw::Bool=true, [auth::Authentication]
 ) -> AbstractJuliaHubApp

Returns the application corresponding to name from the specified category of applications. Will throw an InvalidRequestError if the application can't be found, or returns nothing in this situation if throw=false is passed.

category specifies the application category and must be one of: :default, :package, or :user. This is necessary to disambiguate apps with the same name in the different categories.

See also: applications.

Examples

julia> JuliaHub.applications()
@@ -188,7 +188,7 @@
  JuliaHub.application(:default, "Windows Workstation")
  JuliaHub.application(:package, "RegisteredPackageApp")
  JuliaHub.application(:package, "CustomDashboardApp")
- JuliaHub.application(:user, "ExampleApp.jl")
Experimental API

Applications-related APIs are experimental, and may be changed or removed without notice.

source
JuliaHub.DefaultApp — Type
struct DefaultApp <: AbstractJuliaHubApp

Represents a default JuliaHub instance application, and they can be started as jobs with submit_job.

The list of available applications can be accessed via the applications function, and specific applications can be picked out with application.

julia> apps = JuliaHub.applications(:default)
+ JuliaHub.application(:user, "ExampleApp.jl")
Experimental API

Applications-related APIs are experimental, and may be changed or removed without notice.

source
JuliaHub.DefaultApp — Type
struct DefaultApp <: AbstractJuliaHubApp

Represents a default JuliaHub instance application, and they can be started as jobs with submit_job.

The list of available applications can be accessed via the applications function, and specific applications can be picked out with application.

julia> apps = JuliaHub.applications(:default)
 4-element Vector{JuliaHub.DefaultApp}:
  JuliaHub.application(:default, "Linux Desktop")
  JuliaHub.application(:default, "Julia IDE")
@@ -198,7 +198,7 @@
 julia> JuliaHub.application(:default, "Pluto")
 DefaultApp
  name: Pluto
- key: pluto
Experimental API

Applications-related APIs are experimental, and may be changed or removed without notice.

source
JuliaHub.PackageApp — Type
struct PackageApp <: AbstractJuliaHubApp

Represents a JuliaHub package application that is available in one of the instance's package registries. These packages can be started as JuliaHub jobs with submit_job.

The list of available applications can be accessed via the applications function, and specific applications can be picked out with application.

julia> apps = JuliaHub.applications(:package)
+ key: pluto
Experimental API

Applications-related APIs are experimental, and may be changed or removed without notice.

source
JuliaHub.PackageApp — Type
struct PackageApp <: AbstractJuliaHubApp

Represents a JuliaHub package application that is available in one of the instance's package registries. These packages can be started as JuliaHub jobs with submit_job.

The list of available applications can be accessed via the applications function, and specific applications can be picked out with application.

julia> apps = JuliaHub.applications(:package)
 2-element Vector{JuliaHub.PackageApp}:
  JuliaHub.application(:package, "RegisteredPackageApp")
  JuliaHub.application(:package, "CustomDashboardApp")
@@ -207,14 +207,14 @@
 PackageApp
  name: RegisteredPackageApp
  uuid: db8b4d46-bfad-4aa5-a5f8-40df1e9542e5
- registry: General (23338594-aafe-5451-b93e-139f81909106)

See also: help.juliahub.com on applications

Experimental API

Applications-related APIs are experimental, and may be changed or removed without notice.

source
JuliaHub.UserApp — Type
struct UserApp <: AbstractJuliaHubApp

Represents a private application that has been added to the user account via a Git repository. These applications can be started as JuliaHub jobs with submit_job.

The list of available applications can be accessed via the applications function, and specific applications can be picked out with application.

julia> apps = JuliaHub.applications(:user)
+ registry: General (23338594-aafe-5451-b93e-139f81909106)

See also: help.juliahub.com on applications

Experimental API

Applications-related APIs are experimental, and may be changed or removed without notice.

source
JuliaHub.UserApp — Type
struct UserApp <: AbstractJuliaHubApp

Represents a private application that has been added to the user account via a Git repository. These applications can be started as JuliaHub jobs with submit_job.

The list of available applications can be accessed via the applications function, and specific applications can be picked out with application.

julia> apps = JuliaHub.applications(:user)
 1-element Vector{JuliaHub.UserApp}:
  JuliaHub.application(:user, "ExampleApp.jl")
 
 julia> JuliaHub.application(:user, "ExampleApp.jl")
 UserApp
  name: ExampleApp.jl
- repository: https://github.com/JuliaHubExampleOrg/ExampleApp.jl

See also: help.juliahub.com on applications

Experimental API

Applications-related APIs are experimental, and may be changed or removed without notice.

source
JuliaHub.ApplicationJob — Type
struct ApplicationJob <: AbstractJobConfig

AbstractJobConfig that wraps a DefaultApp. This is primarily used internally and should rarely be constructed explicitly.

source
JuliaHub.PackageJob — Type
struct PackageJob <: AbstractJobConfig

AbstractJobConfig that wraps a PackageApp or UserApp. This is primarily used internally and should rarely be constructed explicitly.

Constructors

JuliaHub.PackageJob(app::Union{JuliaHub.PackageApp,JuliaHub.UserApp}; [sysimage::Bool = false])

Can be used to construct a PackageApp or UserApp based job, but allows for some job parameters to be overridden. Currently, only support the enabling of a system image based job by setting sysimage = true.

julia> app = JuliaHub.application(:package, "RegisteredPackageApp")
+ repository: https://github.com/JuliaHubExampleOrg/ExampleApp.jl

See also: help.juliahub.com on applications

Experimental API

Applications-related APIs are experimental, and may be changed or removed without notice.

source
JuliaHub.ApplicationJob — Type
struct ApplicationJob <: AbstractJobConfig

AbstractJobConfig that wraps a DefaultApp. This is primarily used internally and should rarely be constructed explicitly.

source
JuliaHub.PackageJob — Type
struct PackageJob <: AbstractJobConfig

AbstractJobConfig that wraps a PackageApp or UserApp. This is primarily used internally and should rarely be constructed explicitly.

Constructors

JuliaHub.PackageJob(app::Union{JuliaHub.PackageApp,JuliaHub.UserApp}; [sysimage::Bool = false])

Can be used to construct a PackageApp or UserApp based job, but allows for some job parameters to be overridden. Currently, only support the enabling of a system image based job by setting sysimage = true.

julia> app = JuliaHub.application(:package, "RegisteredPackageApp")
 PackageApp
  name: RegisteredPackageApp
  uuid: db8b4d46-bfad-4aa5-a5f8-40df1e9542e5
@@ -224,9 +224,9 @@
 JuliaHub.Job: jr-xf4tslavut (Submitted)
  submitted: 2023-03-15T07:56:50.974+00:00
  started:   2023-03-15T07:56:51.251+00:00
- finished:  2023-03-15T07:56:59.000+00:00
source

Index

+ diff --git a/dev/reference/jobs/index.html b/dev/reference/jobs/index.html index 7f0d20a76..21dcbab22 100644 --- a/dev/reference/jobs/index.html +++ b/dev/reference/jobs/index.html @@ -13,18 +13,18 @@ } FINISHED:::user -classDef user fill:lightgray
Refreshing a Job object

An instance of a Job object reflects the state of the job when the job function was called. If you want to inspect the current state of a job, you must first "refresh" the job object, which can simply be done with the help of the job function.

job = JuliaHub.job(job)

Reference

JuliaHub.JobReference — Type
const JobReference :: Type

A type constraint on the arguments of many jobs-related functions that is used to specify the job. A job reference must be either a Job object, or an AbstractString containing the unique job ID.

source
JuliaHub.jobs — Function
JuliaHub.jobs(; [limit::Integer], [auth::Authentication]) -> Vector{Job}

Retrieve the list of jobs, latest first, visible to the currently authenticated user.

By default, JuliaHub only returns up to 20 jobs. However, this default limit can be overridden by passing the limit keyword (which must be a positive integer).

Non-dynamic job objects

Job objects represents the jobs when the objects were created (e.g. by a job function) and are not automatically kept up to date. To refresh the job information, you can pass the existing Job to JuliaHub.job before passing it to this function.

source
JuliaHub.job — Function
JuliaHub.job(job::JobReference; throw::Bool=true, [auth::Authentication]) -> Job

Fetch the details of a job based on the job reference ref. Will throw an InvalidRequestError if the job does not exist, or returns nothing if throw=false is passed.

Non-dynamic job objects

Job objects represents the jobs when the objects were created (e.g. by a job function) and are not automatically kept up to date. To refresh the job information, you can pass the existing Job to JuliaHub.job before passing it to this function.

source
JuliaHub.isdone — Function
JuliaHub.isdone(::Job)

A helper function to check if a Job is "done", i.e. its status is one of Completed, Stopped, or Failed.

Non-dynamic job objects

Job objects represents the jobs when the objects were created (e.g. by a job function) and are not automatically kept up to date. As such, the result from this function may not represent the current live state of the job. To refresh the job information, you can pass the existing Job to JuliaHub.job before passing it to this function.

source
JuliaHub.wait_job — Function
wait_job(
+classDef user fill:lightgray
Refreshing a Job object

An instance of a Job object reflects the state of the job when the job function was called. If you want to inspect the current state of a job, you must first "refresh" the job object, which can simply be done with the help of the job function.

job = JuliaHub.job(job)

Reference

JuliaHub.JobReference — Type
const JobReference :: Type

A type constraint on the arguments of many jobs-related functions that is used to specify the job. A job reference must be either a Job object, or an AbstractString containing the unique job ID.

source
JuliaHub.jobs — Function
JuliaHub.jobs(; [limit::Integer], [auth::Authentication]) -> Vector{Job}

Retrieve the list of jobs, latest first, visible to the currently authenticated user.

By default, JuliaHub only returns up to 20 jobs. However, this default limit can be overridden by passing the limit keyword (which must be a positive integer).

Non-dynamic job objects

Job objects represents the jobs when the objects were created (e.g. by a job function) and are not automatically kept up to date. To refresh the job information, you can pass the existing Job to JuliaHub.job before passing it to this function.

source
JuliaHub.job — Function
JuliaHub.job(job::JobReference; throw::Bool=true, [auth::Authentication]) -> Job

Fetch the details of a job based on the job reference ref. Will throw an InvalidRequestError if the job does not exist, or returns nothing if throw=false is passed.

Non-dynamic job objects

Job objects represents the jobs when the objects were created (e.g. by a job function) and are not automatically kept up to date. To refresh the job information, you can pass the existing Job to JuliaHub.job before passing it to this function.

source
JuliaHub.isdone — Function
JuliaHub.isdone(::Job)

A helper function to check if a Job is "done", i.e. its status is one of Completed, Stopped, or Failed.

Non-dynamic job objects

Job objects represents the jobs when the objects were created (e.g. by a job function) and are not automatically kept up to date. As such, the result from this function may not represent the current live state of the job. To refresh the job information, you can pass the existing Job to JuliaHub.job before passing it to this function.

source
JuliaHub.wait_job — Function
wait_job(
     job::AbstractString;
     interval::Integer = 30, [auth::Authentication]
-) -> Job

Blocks until remote job referred to by the job reference job has completed, by polling it with every interval seconds. Returns an updated Job object.

source
JuliaHub.extend_job — Function
JuliaHub.extend_job(job::JobReference, extension::Limit; [auth::Authentication]) -> Job

Extends the time limit of the job referred to by the job reference ref by extension (Dates.Period, or Integer number of hours). Returns an updated Job object.

See Limit for more information on how the extension argument is interpreted. Note that Unlimited is not allowed as extension.

See also: Job.

source
JuliaHub.JobLogMessage — Type
struct JobLogMessage

Contains a single JuliaHub job log message, and has the following fields:

  • timestamp :: Union{ZonedDateTime, Nothing}: log message timestamp (in UTC)
  • message :: Union{String, Nothing}: log message string. This generally corresponds to one line of printed output

Fields that can also be nothing may be missing for some log messages.

See also: job_logs, job_logs_buffered.

No public constructors

Objects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.

source
JuliaHub.job_logs — Function
JuliaHub.job_logs(job; offset::Integer = 0, [limit::Integer], [auth::Authentication]) -> Vector{JobLogMessage}

Fetches the log messages for the specified JuliaHub job. The job is specifed by passing the job name as a string, or by passing a Job object (i.e. job::Union{AbstractString,Job}). Returns the log messages as an array of JobLogMessage objects.

Optionally, the function takes the following keyword arguments:

  • offset::Integer: the offset of the first log message fetched (0 corresponds to the first message); for the first method, this defaults to 0; however, in the second (callback) case, if offset is not specified, any existing logs will be ignored.

  • limit::Integer: the maximum number of messages fetched (all by default)

No default limit

The limit keyword does not have a default limit, and so by default job_logs fetches all the log messages. This may take a while and require many requests to JuliaHub if the job has a huge number of log messages.

source
JuliaHub.extend_job — Function
JuliaHub.extend_job(job::JobReference, extension::Limit; [auth::Authentication]) -> Job

Extends the time limit of the job referred to by the job reference ref by extension (Dates.Period, or Integer number of hours). Returns an updated Job object.

See Limit for more information on how the extension argument is interpreted. Note that Unlimited is not allowed as extension.

See also: Job.

source
JuliaHub.JobLogMessage — Type
struct JobLogMessage

Contains a single JuliaHub job log message, and has the following fields:

  • timestamp :: Union{ZonedDateTime, Nothing}: log message timestamp (in UTC)
  • message :: Union{String, Nothing}: log message string. This generally corresponds to one line of printed output

Fields that can also be nothing may be missing for some log messages.

See also: job_logs, job_logs_buffered.

No public constructors

Objects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.

source
JuliaHub.job_logs — Function
JuliaHub.job_logs(job; offset::Integer = 0, [limit::Integer], [auth::Authentication]) -> Vector{JobLogMessage}

Fetches the log messages for the specified JuliaHub job. The job is specifed by passing the job name as a string, or by passing a Job object (i.e. job::Union{AbstractString,Job}). Returns the log messages as an array of JobLogMessage objects.

Optionally, the function takes the following keyword arguments:

  • offset::Integer: the offset of the first log message fetched (0 corresponds to the first message); for the first method, this defaults to 0; however, in the second (callback) case, if offset is not specified, any existing logs will be ignored.

  • limit::Integer: the maximum number of messages fetched (all by default)

No default limit

The limit keyword does not have a default limit, and so by default job_logs fetches all the log messages. This may take a while and require many requests to JuliaHub if the job has a huge number of log messages.

source
JuliaHub.job_logs_buffered — Function
JuliaHub.job_logs_buffered(
     [f::Base.Callable], job::Union{Job,AbstractString};
     streaming::Bool=true, [offset::Integer],
     [auth::Authentication]
-) -> AbstractJobLogsBuffer

A lower-level function to work with log streams, and is particularly useful when working with jobs that have not finished yet and are actively producing new log messages.

The function accepts the following arguments:

  • f :: Base.Callable: an optional callback function that gets called every time the buffer is updated. The callback must take two arguments: f(::AbstractJobLogsBuffer, ::AbstractVector). The first argument is the buffer object itself, and the second argument will be passed a read-only view of all the logs that have been loaded into the buffer, including the new ones.
  • job :: Union{Job,AbstractString}: either the job name or a Job object.
  • streaming :: Bool: if set to true, the buffer object The streaming can be stopped with interrupt!.
  • offset :: Integer: optional non-negative value to specify the starting point of the buffer

Interface of the returned object

Returns an instance of the abstract AbstractJobLogsBuffer type. These objects contain log messages (of type JobLogMessage), but not all the log messages are immediately available. Instead, at any given time the buffer represents a continous section of logs that can be extended in either direction.

The following functions can be used to interact with log buffers: job_logs_newer!, job_logs_older!, JuliaHub.hasfirst, JuliaHub.haslast. Additionally, the objects will have a .logs :: Vector{JobLogMessage} property that can be used to access the log messages that have been loaded into the buffer.

See also: job_logs, Job.

source
JuliaHub.job_logs_older! — Function
JuliaHub.job_logs_older!(
+) -> AbstractJobLogsBuffer

A lower-level function to work with log streams, and is particularly useful when working with jobs that have not finished yet and are actively producing new log messages.

The function accepts the following arguments:

  • f :: Base.Callable: an optional callback function that gets called every time the buffer is updated. The callback must take two arguments: f(::AbstractJobLogsBuffer, ::AbstractVector). The first argument is the buffer object itself, and the second argument will be passed a read-only view of all the logs that have been loaded into the buffer, including the new ones.
  • job :: Union{Job,AbstractString}: either the job name or a Job object.
  • streaming :: Bool: if set to true, the buffer object The streaming can be stopped with interrupt!.
  • offset :: Integer: optional non-negative value to specify the starting point of the buffer

Interface of the returned object

Returns an instance of the abstract AbstractJobLogsBuffer type. These objects contain log messages (of type JobLogMessage), but not all the log messages are immediately available. Instead, at any given time the buffer represents a continous section of logs that can be extended in either direction.

The following functions can be used to interact with log buffers: job_logs_newer!, job_logs_older!, JuliaHub.hasfirst, JuliaHub.haslast. Additionally, the objects will have a .logs :: Vector{JobLogMessage} property that can be used to access the log messages that have been loaded into the buffer.

See also: job_logs, Job.

source
JuliaHub.job_logs_older! — Function
JuliaHub.job_logs_older!(
     buffer::AbstractJobLogsBuffer; [count::Integer], [auth::Authentication]
-) -> AbstractJobLogsBuffer

Updates the AbstractJobLogsBuffer object by adding up to count log messages to the beginning of the buffer. If count is omitted, it will seek all the way to the beginning of the logs.

If all the logs have already been loaded into the buffer (i.e. JuliaHub.hasfirst(buffer) is true), the function is a no-op.

See also: job_logs_buffered, job_logs_newer!.

source
JuliaHub.job_logs_newer! — Function
JuliaHub.job_logs_newer!(
+) -> AbstractJobLogsBuffer

Updates the AbstractJobLogsBuffer object by adding up to count log messages to the beginning of the buffer. If count is omitted, it will seek all the way to the beginning of the logs.

If all the logs have already been loaded into the buffer (i.e. JuliaHub.hasfirst(buffer) is true), the function is a no-op.

See also: job_logs_buffered, job_logs_newer!.

source
JuliaHub.job_logs_newer! — Function
JuliaHub.job_logs_newer!(
     buffer::AbstractJobLogsBuffer; [count::Integer], [auth::Authentication]
-) -> AbstractJobLogsBuffer

Updates the AbstractJobLogsBuffer object by adding up to count log messages to the end of the buffer. If count is omitted, it will seek all the way to the end of the current logs.

For a finished job, if all the logs have already been loaded into the buffer (i.e. JuliaHub.haslast(buffer) is true), the function is a no-op. If the buffer is actively streaming new logs for a running job, then the function is also a no-op.

See also: job_logs_buffered, job_logs_older!.

source
JuliaHub.haslast — Function
JuliaHub.haslast(::AbstractJobLogsBuffer) -> Bool

Determines whether the job log buffer has the last message of the job logs. Note that if the job has not finished, this will always be false, since the job may produce additional logs.

See also: hasfirst, job_logs_buffered.

source
JuliaHub.interrupt! — Function
JuliaHub.interrupt!(::AbstractJobLogsBuffer; wait::Bool=true)

Can be use to interrupt the asynchronous log streaming task. If the log buffer is not streaming, this function is a no-op.

Note that the effect of JuliaHub.interrupt! may not be immediate and the function will block until the task has stopped. wait = false can be passed to make interrupt! return immediately, but in that case the buffer may stream for a little while longer.

source
JuliaHub.job_files — Function
JuliaHub.job_files(job::Job, [filetype::Symbol]) -> Vector{JobFile}

Return the list of inputs and/or output files associated job.

The optional filetype argument should be one of :input, :source, :result or :project, and can be used to filter the file list down to either just job input files (such as the appbundle or Julia environment files), or output files (such as the one uploaded via RESULTS_FILE).

Note: job_file(job) is equivalent to job.files, and the latter is preferred. This function is primarily meant to be used when filtering by file type.

See also: Job.

source
JuliaHub.job_file — Function
JuliaHub.job_file(job::Job, type::Symbol, filename::AbstractString) -> JobFile | Nothing

Searches for a job output file of a specified type and with the specific filename for job job, or nothing if the file was not found.

type should be one of the standard job file types. See JobFile and job_files for more information.

source
JuliaHub.download_job_file — Function
JuliaHub.download_job_file(file::JobFile, path::AbstractString; [auth]) -> String
+) -> AbstractJobLogsBuffer

Updates the AbstractJobLogsBuffer object by adding up to count log messages to the end of the buffer. If count is omitted, it will seek all the way to the end of the current logs.

For a finished job, if all the logs have already been loaded into the buffer (i.e. JuliaHub.haslast(buffer) is true), the function is a no-op. If the buffer is actively streaming new logs for a running job, then the function is also a no-op.

See also: job_logs_buffered, job_logs_older!.

source
JuliaHub.haslast — Function
JuliaHub.haslast(::AbstractJobLogsBuffer) -> Bool

Determines whether the job log buffer has the last message of the job logs. Note that if the job has not finished, this will always be false, since the job may produce additional logs.

See also: hasfirst, job_logs_buffered.

source
JuliaHub.interrupt! — Function
JuliaHub.interrupt!(::AbstractJobLogsBuffer; wait::Bool=true)

Can be use to interrupt the asynchronous log streaming task. If the log buffer is not streaming, this function is a no-op.

Note that the effect of JuliaHub.interrupt! may not be immediate and the function will block until the task has stopped. wait = false can be passed to make interrupt! return immediately, but in that case the buffer may stream for a little while longer.

source
JuliaHub.job_files — Function
JuliaHub.job_files(job::Job, [filetype::Symbol]) -> Vector{JobFile}

Return the list of inputs and/or output files associated job.

The optional filetype argument should be one of :input, :source, :result or :project, and can be used to filter the file list down to either just job input files (such as the appbundle or Julia environment files), or output files (such as the one uploaded via RESULTS_FILE).

Note: job_file(job) is equivalent to job.files, and the latter is preferred. This function is primarily meant to be used when filtering by file type.

See also: Job.

source
JuliaHub.job_file — Function
JuliaHub.job_file(job::Job, type::Symbol, filename::AbstractString) -> JobFile | Nothing

Searches for a job output file of a specified type and with the specific filename for job job, or nothing if the file was not found.

type should be one of the standard job file types. See JobFile and job_files for more information.

source
JuliaHub.download_job_file — Function
JuliaHub.download_job_file(file::JobFile, path::AbstractString; [auth]) -> String
 JuliaHub.download_job_file(file::JobFile, io::IO; [auth])

Downloads a JobFile to a local path. Alternative, writeable stream object can be passed as the second argument to write the contents directly into the stream.

When a local path is passed, it returns the path (which can be useful when calling the function as e.g. JuliaHub.download_job_file(file, tempname()))). When an IO object is passed, it returns nothing.

For example, to download a file into a temporary file:

julia> file = JuliaHub.job_file(JuliaHub.job("jr-eezd3arpcj"), :result, "outdir.tar.gz")
 JuliaHub.JobFile outdir.tar.gz (jr-eezd3arpcj, :result, 632143 bytes)
 sha2_256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
@@ -42,7 +42,7 @@
 julia> JuliaHub.download_job_file(file, buffer)
 
 julia> bytes2hex(sha2_256(take!(buffer)))
-"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"

See also: Job, JobFile.

source
JuliaHub.Job — Type
struct Job

Represents a single job submitted to JuliaHub. Objects have the following properties:

  • id :: String: the unique, automatically generated ID of the job
  • alias :: String: a non-unique, but descriptive alias for the job (often set by e.g. applications)
  • status :: JobStatus: a string-like JobStatus object storing the state of the job
  • env :: Dict: a dictionary of environment variables that were set when the job was submitted
  • results :: String: the output value set via ENV["RESULTS"] (an empty string if it was not explicitly set)
  • files :: Vector{JobFiles}: a list of JobFile objects, representing the input and output files of the job (see: job_files, job_file, download_job_file).

See also: job, jobs.

Non-dynamic job objects

Job objects represents the jobs when the objects were created (e.g. by a job function) and are not automatically kept up to date. To refresh the job information, you can pass the existing Job to JuliaHub.job before passing it to this function.

No public constructors

Objects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.

source
JuliaHub.JobStatus — Type
struct JobStatus

Type of the .status field of a Job object, representing the current state of the job. Should be one of: Submitted, Running, Failed, Stopped, Completed.

In practice, the .status field should be treated as string and only used in string comparisons.

See also: isdone.

julia> job = JuliaHub.job("jr-novcmdtiz6")
+"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"

See also: Job, JobFile.

source
JuliaHub.Job — Type
struct Job

Represents a single job submitted to JuliaHub. Objects have the following properties:

  • id :: String: the unique, automatically generated ID of the job
  • alias :: String: a non-unique, but descriptive alias for the job (often set by e.g. applications)
  • status :: JobStatus: a string-like JobStatus object storing the state of the job
  • env :: Dict: a dictionary of environment variables that were set when the job was submitted
  • results :: String: the output value set via ENV["RESULTS"] (an empty string if it was not explicitly set)
  • files :: Vector{JobFiles}: a list of JobFile objects, representing the input and output files of the job (see: job_files, job_file, download_job_file).

See also: job, jobs.

Non-dynamic job objects

Job objects represents the jobs when the objects were created (e.g. by a job function) and are not automatically kept up to date. To refresh the job information, you can pass the existing Job to JuliaHub.job before passing it to this function.

No public constructors

Objects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.

source
JuliaHub.JobStatus — Type
struct JobStatus

Type of the .status field of a Job object, representing the current state of the job. Should be one of: Submitted, Running, Failed, Stopped, Completed.

In practice, the .status field should be treated as string and only used in string comparisons.

See also: isdone.

julia> job = JuliaHub.job("jr-novcmdtiz6")
 JuliaHub.Job: jr-novcmdtiz6 (Completed)
  submitted: 2023-03-15T07:56:50.974+00:00
  started:   2023-03-15T07:56:51.251+00:00
@@ -58,9 +58,9 @@
 false
 
 julia> job.status == "Completed"
-true
No public constructors

Objects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.

source
JuliaHub.JobFile — Type
struct JobFile

A reference to a job input or output file, with the following properties:

  • .name :: String: the name of the Job this file is attached to
  • .type :: Symbol: indicated the file type (see below)
  • .filename :: String: file name
  • .size :: Int: size of the file in bytes (reported to be zero in cases where the file contents is missing)
  • .hash :: Union{FileHash, Nothing}: a FileHash object containing the file hash, but may also be missing (nothing) in some cases, like when the file upload has not completed yet.

The file is uniquely identified by the (job, type, filename) triplet.

The type of the file should be one of:

  • :input: various job input files, which includes code, environment, and appbundle files.
  • :source: source
  • :project: Julia project environment files (e.g. Project.toml, Manifest.toml, Artifacts.toml)
  • :result: output results file, defined via the RESULTS_FILE environment variable in the job

Note that some files are duplicated under multiple categories.

See also: Job, job_files, job_file.

No public constructors

Objects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.

source
JuliaHub.FileHash — Type
struct FileHash

Stores a hash and the algorithm used to calcute it. The object has the following properties:

  • .algorithm :: Symbol: hash algorithm
  • .hash :: Vector{UInt8}: the hash as a sequence of bytes
No public constructors

Objects of this type should not be constructed explicitly. The contructor methods are not considered to be part of the public API.

source

Index

+