Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document using AWS SSO credentials #1075

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Document using AWS SSO credentials #1075

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
444a452
document using AWS (CLI, python, R) with SSO credentials
dtenenba Oct 18, 2024
7fed982
give up on spoiler functionality
dtenenba Oct 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 55 additions & 0 deletions _compdemos/aws-s3.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -256,6 +256,61 @@ s3.download_file(bucket_name, "df.csv", "df.csv")

# AWS via R

To use AWS from R with [AWS SSO credentials](/scicomputing/access_credentials/#amazon-web-services-aws),
you will need to first login to an SSO session. This is not necessary when using the AWS CLI or Python.
Note that the `aws sso login` step will require you to copy a URL into your browser and paste a code back into the terminal. Once you've done that, you have a session that will last for 12 hours.

```
ml purge
ml awscli
aws sso login
ml fhR
R
```

Then within R, you can use the `aws.s3` or `paws` packages to interact with S3.
`paws` will "just work" out of the box. To use `aws.s3` you will need to run this code
first:

```r
# Load required libraries
library(jsonlite)
library(lubridate)

# Define the path to the AWS CLI cache directory
cache_dir <- "~/.aws/cli/cache/"

# Get the list of files in the cache directory
cache_files <- list.files(cache_dir, full.names = TRUE)

# Find the most recently modified file
latest_file <- cache_files[which.max(file.info(cache_files)$mtime)]

# Read the JSON content from the latest file
json_content <- fromJSON(latest_file)

# Extract the relevant credentials
access_key <- json_content$Credentials$AccessKeyId
secret_key <- json_content$Credentials$SecretAccessKey
session_token <- json_content$Credentials$SessionToken
expiration<- as.character(with_tz(ymd_hms(json_content$Credentials$Expiration)))
region <- "us-west-2"

# Set the environment variables using the extracted credentials
Sys.setenv(
AWS_ACCESS_KEY_ID = access_key,
AWS_SECRET_ACCESS_KEY = secret_key,
AWS_SESSION_TOKEN = session_token,
AWS_DEFAULT_REGION = region
)

# Print a message to confirm that the environment variables have been set
cat("AWS credentials have been set from the most recent SSO cache file.\n")
cat("They will be valid until " , expiration, "\n")
```



You can use [Amazon Web Services' S3](https://aws.amazon.com/s3/) (Simple Storage Service) directly from `R`. The `R` package which facilitates this, `aws.s3`, is included in recent builds of `R` available on the `rhino` systems and the `gizmo` cluster.

## Getting Started
Expand Down
46 changes: 37 additions & 9 deletions _scicomputing/access_credentials.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,25 +47,53 @@ AWS credentials are designated per user, so each Fred Hutch employee should obta

>Note: Beyond precautions taken to protect any other credentials listed here, take care to ensure AWS credentials are never shared with or disclosed to any other user, directly (e.g., by email) or indirectly (e.g., by including them in code and sharing the code/committing to GitHub). If you need credentials for an external collaborator, or if you are having a permissions issue, please email `helpdesk` to request support from [Scientific Computing](https://centernet.fredhutch.org/cn/u/center-it/cio/scicomp.html).

You will receive your AWS credentials via an encrypted email when you are onboarded, or if you need to request credentials for an existing employee, please email `helpdesk`.
To get your AWS credentials, visit the [MyApps](https://myapps.microsoft.com) dashboard
and click the square entitled `AWS IAM Identity Center - FHCC-H`. Sign in with your HutchNet ID and password.

This will take you to a screen called `AWS accounts`. You should see your accunt listed.
For example, if your PI is Jane Doe, you should see `fh-pi-doe-j` listed.
Click the triangle to the left of the account name. Now you'll see two links.
The link on the left will take you to the AWS console, which is web
browser interface to Amazon Web Services. The link on the right,
`Access keys` will give you the credentials you need to use AWS outside
of a browser.

The next section will describe how to configure the AWS CLI with these credentials.

Once you have working credentials, you can read more about [AWS Storage](/scicomputing/store_objectstore/) and [AWS Computing](/scicomputing/compute_cloud/) in our wiki pages.

### Configure AWS CLI

Load the `awscli` module, then run `aws configure` and enter your Access Key ID & Secret Access Key. You can read more about access key creation/modification [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey).
You should be on the `Get credentials` page as described in the previous section,
and you should have a terminal window connected to one of the `rhino` machines via [ssh](https://sciwiki.fredhutch.org/scicomputing/access_methods/#ssh-clients-for-remote-computing-resources).

Load the `awscli` module (with the `ml awscli` command), then run `aws configure sso`.
For `SSO session name` you can enter any string. For `SSO start URL`, enter the `SSO start URL` shown in your browser. For `SSO region`, enter `us-west-2`. For `SSO registration scopes`, press Enter.

You will now see a URL and a code displayed. Copy and paste the URL into your browser, and enter the code on the resulting page. Click `Allow Access`.

If you have access to more than one AWS account, you should now choose the same account
you choose in the last step, then press Enter.
For `CLI default client Region`, press Enter. For `CLI default output format`, press Enter.

The next and final piece of information to fill in is the `CLI profile name`.
If you have not set up AWS credentials before, you should use the value `default`.

The terminal will now display the following:

```
module load awscli
aws configure
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-west-2
Default output format [None]:
To use this profile, specify the profile name using --profile, as shown:

aws s3 ls --profile default
```
This will create the following files that store your credentials `~/.aws/config` & `~/.aws/credentials`

The `--profile default` flag is not necessary if you are using the default profile.

The following section will describe how to test and use your credentials.


### Testing Your Credentials

To test your credentials to ensure that you have the correct permissions to your PI bucket, execute the following to copy a file from your local computer to your PI's bucket.

In these examples, please replace `lastname-f` with the last name and first initial of your PI.
Expand Down