v1.0rc2 testing (v1.0rc2-091f09f4 on OSX Catalina / Chrome 90.0.4430.85)

[bodom0015]

Test Plan

https://github.com/whole-tale/wt-design-docs/issues/new?template=TEST_PLAN.md

Note: For all tests, repeat for supported browser/OS combinations.

Preconditions:

• Clear browsing history or open incognito/private session
• Disconnect all third party integrations

Splash page

• Splash page

1. Goto https://dashboard.stage.wholetale.org
2. Confirm Terms of Service link
3. Confirm acknowledgements (NSF, XSEDE/Jetstream, Stata)
4. Confirm partner logos

Authentication

• Basic login flow

1. Goto https://dashboard.stage.wholetale.org
2. Select "Access Whole Tale" button. You should be prompted to login via Globus Auth.
3. Select your organization and login
4. You should be taken to the "My Tales" page
5. Your Gravatar should display in the upper right

• Basic logout flow

1. Select the "User" menu
2. Select the "Log out" menu item. You should be taken to the login page.
3. Select "Access Whole Tale" button.
4. Note that you are not prompted to login.
5. Restart your incognito session
6. Select "Access Whole Tale" button. You should be prompted to login via Globus Auth.

• Return-route for non-logged in users

1. Logout from Whole Tale
2. Attempt to access https://dashboard.stage.wholetale.org/public
3. You should be redirected to https://dashboard.stage.wholetale.org/login?rd=%252Fpublic
4. Select "Access Whole Tale".
5. You should be redirected to the "Public Tales" page.

Navigation

• General navigation

1. Select "Tale Dashboard" link. "My Tales" page should display
2. Click "Create New Tale" > "Create New Tale". The Create Tale modal should display. Click "Cancel" to close the modal
3. Select the "i" icon at the upper right corner; it should open the User Guide in new tab
4. Select the "Notification" button next to the "i" icon; it should display the notification panel
5. Select the "Report a problem" button; it should open https://github.com/whole-tale/whole-tale/issues in new tab
6. Mouse over a Tale and select "View". The Run page should display
7. Click the "user" icon at the top right

• You should see a dropdown expand
• The dropdown should contain the user's first/last name and gravatar image
• The dropdown should provide a link to the Settings page
• The dropdown should provide an option to Logout
• Select the "Settings" option; it should bring you to the Settings page

Tale Dashboard

Preconditions:

• Assumes production Tales present (e.g., LIGO, materials, etc).

• No running instances

• General

1. Select "My Tales" link. Confirm "My Tales" page displays
2. Confirm that no Tales are running
3. Select "Shared with Me" link. Confirm "Shared with Me" page displays

• Search

1. Select "Public Tales" link. Confirm "Public Tales" page displays
2. Enter "Ligo" as search term. Confirm only LIGO tale displays (by K. Kowalik)
3. Clear search term. Confirm all tales display

• View tale

1. Select "View" button on LIGO Tale
2. Confirm that Metadata page displays by default
3. Confirm that you cannot delete or edit Tale properties
4. Select Back button. Confirm that you are taken back to My Tales page
5. Select "View" button on Tale you own
   1. Confirm that the Run and Close buttons are present
   2. Confirm that you can edit Tale properties
   3. Select "Close"
   4. Confirm that you are taken back to "Public Tales"

FAIL: Interact page displayed by default, instead of Metadata

Currently, clicking on View leads to interact, while clicking the Tale's title will bring you to metadata. This is probably too obscure / confusing, so I can change these to point to metadata and match the test case.

Also noting that this is inconsistent with "Running Tales", where the two destinations are swapped. I think it definitely makes sense for this to be consistent across cards, at the very least.

• Launch instance

1. Select "Run Tale" button on "Ligo" tale
   1. Confirm you are prompted to Copy and Run
   2. Select "Confirm"
   3. Confirm that you are redirected to the Tale's "Metadata" page
   4. Confirm that the "Edit" button is present
   5. Select "Run Tale"
   6. Confirm that the "Interact" page is displayed with spinner
   7. Confirm that the notification dialog appears
   8. Confirm that the step progression is correct
   9. Confirm that the "View Logs" button shows the build logs in a modal
   10. Navigate to the "My Tales" page
   11. Confirm that the Tale is shown under "Currently Running"
   12. Select "View" on the Tale card
2. Select "Stop" to end the Tale instance
   1. Confirm that the Tale has stopped
   2. Confirm that the button has changed to Run
3. Navigate to Browse
   1. Confirm that there are no Tales under "Currently Running"
   2. Select the "Run Tale" button on a Tale you own
   3. After the instance is launched, navigate back to the Browse page
   4. Try to delete the Tale (not instance)
   5. Confirm you cannot select the delete button on the browse page
   6. Confirm that the "Run Tale" button has been replaced with "Stop Tale"
   7. Confirm that the Tale is shown at the top of the page under "Currently running"

FAIL: Redirected to Interact, instead of Metadata (likely due to previous test failure)

Noting that copyOnLaunch does redirect to the run view, but does not explicitly specify a tab. We can add a queryParam here to specify an explicit tab destination, if it would make sense. Interact seems like the sensible choice to me, since a user would only encounter copyOnLaunch when trying to actually Run the Tale, which for me is enough to indicate that a user wants to Interact with it.

Managing Data

Preconditions

• Empty home directory

• No registered data

• Register General

1. Open an owned Tale
2. Open the data registration panel from the 'External Data' tab
3. Confirm list of providers matches list of supported providers (Globus, DataONE, Dataverse, Zenodo)

• Register DataONE data

1. Open an owned Tale
2. Open the data registration panel from the 'External Data' tab
3. Confirm "Search for Data Sources" modal displays
4. Search for DOI 10.5065/D6862DM8
5. Confirm "Humans and Hydrology" dataset is found on DataONE.
6. Select Register
7. Confirm dataset appears in Manage Data list
8. Confirm additional URL formats:
   1. https://search.dataone.org/view/doi:10.5063/F1Z60M87
   2. https://search.dataone.org/view/https://pasta.lternet.edu/package/metadata/eml/knb-lter-nwt/115/1
   3. https://search.dataone.org/view/doi:10.18739/A2CN6XZ7H
   4. https://raw.githubusercontent.com/whole-tale/dashboard/master/.travis.yml

FAIL: Dataset name is incorrect, data appears invalid (none of these 5 produced correct results for me on stage)

• Dataverse

1. Register dataset doi:10.7910/DVN/SZ9YXZ
2. Confirm "Replication Data for: Direct Democracy and Women's Political Engagement" from Dataverse

• Globus/MDF

1. Confirm DOI resolution for http://dx.doi.org/doi:10.18126/M2301J results in https://petreldata.net/mdf/detail/pub_30_shahani_twinmediated_v1.2
2. Register dataset https://petreldata.net/mdf/detail/pub_30_shahani_twinmediated_v1.2

FAIL: Found 0 results

Run Tale

Preconditions:

• No running Tale instances

• General

1. Launch (Copy and Run) the Ligo Tale and wait for the instance to start
2. Confirm you are redirected to the Run page
   1. Confirm that the Run panel displays the Tale icon, title, and author
   2. Confirm that the Run page displays the Interact, Files, Metadata tabs
3. Select the "..." menu.
   1. Confirm that menu options exist for "Rebuild Tale", "Restart Tale", "Save Tale Version", "Duplicate Tale", "Publish Tale", "Export Tale", "Connect to Git Repository..."
   2. Confirm that the "Read the docs" link displays https://wholetale.readthedocs.io/en/stable/users_guide/run.html in a new tab or window
   3. Confirm that the "View Fullscren" link displays the Run panel in full screen

TEST: latest now used, instead of stable

• Interact tab

1. Select the "Interact" tab
2. Confirm Jupyter environment is displayed in iframe
3. Select "popout" icon. Confirm Jupyter is opened in new Tab

• Files tab

1. Select "Files" tab
2. You should see a vertical navigation menu with three tabs: "Home", "External Data", and "Tale Workspace".

• Metadata tab - Owned Tale

1. Open a Tale that you own
2. Navigate to the metadata page
3. Add multiple datasets that have DOIs to the Tale
4. Navigate to Run > Metadata
5. Valid environments should be listed in the Environment dropdown menu
6. Valid licenses should be listed in the License dropdown
7. Confirm that you see Created by <your_name> under Authors
8. Confirm that your Tale has the CC4 license assigned to it
9. Confirm that the Tale's Published Location reads This Tale has not been published
10. Confirm that there are citations for the external datasets
11. Click the "Edit" button to edit the metadata fields
12. Change the name of the Tale, Category, Description, Illustration, Public, License, and add multiple authors
13. Select Save. Confirm that you see a notification telling you that the Tale was saved.
14. Refresh the page. Your changes should be preserved between page refreshes, confirming that they are being properly persisted.
15. Confirm that the new Tale authors are displayed on the Tale card

TEST: Reorder test steps to reflect reality - e.g. need to click edit before seeing dropdowns, "Created By" no longer listed under authors (see whole-tale/ngx-dashboard#157), etc

• Metadata tab - Non-Owned Tale

1. Open a Tale that you don't own
2. Navigate to the metadata page
3. Note that there isn't an 'Edit' button present

• Home

1. Select Files > Home. Confirm that your home directory displays
2. Create a folder and upload a file to the folder
3. Confirm file operations:
   1. Copy, rename, remove, download, Move To...

FAIL: Copy does not work properly if file is inside of a subfolder of Home folder (copy is placed in the root of the Home folder, UI state is mismatched and invalid)
FAIL: minor note that I still encounter whole-tale/ngx-dashboard#8

• External data

1. Precondition: Register one or more Datasets (e.g. doi:10.18739/A29G5GD0V)
   1. Confirm notification progress and success
2. Confirm "External Data" list is initially empty
3. Confirm (+) button ("Select Data modal") behavior
   1. Confirm that "WT Data Catalog" shows all registered datasets
   2. Confirm that "My External Data" filters to only show your registered datasets
   3. Select a dataset
   4. Confirm that clicking "Add Selected" adds the dataset to the "Selected Data" column
   5. Confirm that "Remove Selected" data to dissociate it with a Tale
   6. Using the dataset above, add the dataset to the Tale, by clicking "Save"
   7. Confirm that the Tale's "External Data" list matches what was saved
   8. Remove the dataset from the Tale
   9. Confirm "External Data" list matches what was saved
   10. Select a dataset but then click out of the modal or click "Cancel"
   11. Confirm that the changes weren't saved

• Workspaces

1. Confirm create folder
2. Confirm upload file
3. Select "+" button has options "New folder", "Upload file", Import Tale Data..."
4. Folder operations
   1. Confirm context menu Copy, Rename, Download, Remove
   2. Confirm rename folder
   3. Confirm download folder (this should result in a zip file)
   4. Confirm remove folder
5. File operations
   1. Upload a file
   2. Confirm context menu Copy, Rename, Download, Remove
   3. Confirm rename folder
   4. Confirm download file
   5. Confirm remove file
6. Select "Select Data..."
   1. Note vertical navigation with Tale Workspaces
   2. Copy to Workspace from another workspace
   3. Move to workspace from another workspace

FAIL: Copy folder fails with a 500: {"message": "Folder 'folder' already exists at /tmp/data/workspaces/5/5bdb63616141bc0001cc1b59/folder", "type": "girder"}
TEST: No test steps to exercise File/Folder "Move-To..."
FAIL: UI not automatically refreshed after "copy to workspace" or "move to workspace" completed

• Files - Non-Owned Tale

1. Open a Tale that you don't have write permissions to
2. Attempt to add data to the workspace and external data
3. Note that the "+" icons are greyed out, preventing you from doing so

• Export Tale

1. View a Tale that you own
2. Select "Export Tale" from Tale menu
3. Confirm that a download starts for the ZIP archive
4. Open the archive
5. Confirm bagit format and contents
6. Run the exported tale and confirm that it builds and runs as expected

FAIL: Export worked, but running locally appeared to hang and never finished:

#18 17.81 
#18 17.81 Preparing transaction: ...working... done
#18 18.15 Executing transaction: ...working... WARNING conda.core.envs_manager:register_env(52): Unable to register environment. Path not writable or missing.
#18 20.27   environment location: /srv/conda
#18 20.27   registry file: /root/.conda/environments.txt
#18 20.27 done
#18 20.77 installation finished.
#18 20.78 time: 14.719
#18 20.78 + export PATH=/srv/conda/bin:/srv/conda/envs/notebook/bin:/srv/conda/bin:/srv/npm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
#18 20.78 + PATH=/srv/conda/bin:/srv/conda/envs/notebook/bin:/srv/conda/bin:/srv/npm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
#18 20.78 + conda config --system --append channels defaults
#18 20.91 + conda config --system --set auto_update_conda false
#18 21.06 + conda config --system --set show_channel_urls true
#18 21.23 + echo 'update_dependencies: false'
#18 21.23 + conda config --system --set channel_priority flexible
#18 21.36 + conda install -y mamba==0.6.1
#18 21.73 Collecting package metadata (current_repodata.json): ...working... done
#18 29.92 Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
#18 29.92 Collecting package metadata (repodata.json): ...working... #18 29.92 Collecting package metadata (repodata.json): ...working... 
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/repo2docker/__main__.py", line 411, in main
    r2d.start()
  File "/usr/lib/python3.8/site-packages/repo2docker/app.py", line 778, in start
    self.build()
  File "/usr/lib/python3.8/site-packages/repo2docker/app.py", line 764, in build
    raise docker.errors.BuildError(l["error"], build_log="")
docker.errors.BuildError: #18 29.92 Collecting package metadata (repodata.json): ...working... 

Settings

• Default State

1. Navigate to the Settings page
2. Confirm there are empty settings for DataONE, Dataverse, and Zenodo

• Connect to Zenodo

1. Click "Connect Account" in the Zenodo section
2. Confirm that the connection modal opens
3. Confirm that sandbox.zenodo.org and zenodo.org are listed
4. Enter your API key for one of the repositories
5. Click "Connect"
6. Confirm that you see "Authorized on "
7. Select disconnect, confirmation window should appear
8. Repeat test with invalid key, confirm error message appears
9. Repeat for each Zenodo server

• Connect to Dataverse

1. Click "Connect Account" in the Dataverse section
2. Confirm that the connection modal opens
3. Enter your API key for one of the repositories
4. Click "Connect"
5. Confirm that you see "Authorized on "
6. Select disconnect, confirmation window should appear
7. Repeat test with invalid key, confirm error message appears
8. Repeat for each Dataverse server

FAIL: Unable to login/signup for demo.dataverse.org
FAIL: The following request timed out: POST https://girder.stage.wholetale.org/api/v1/account/dataverse/key?resource_server=dev2.dataverse.org&key=MY_REAL_API_KEY_VALUE&key_type=apikey:

{"message": "ConnectionError: ConnectionError(MaxRetryError(\"HTTPSConnectionPool(host='dev2.dataverse.org', port=443): Max retries exceeded with url: /api/users/token (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f8bcc566c40>: Failed to establish a new connection: [Errno 110] Connection timed out'))\"))", "trace": ["<FrameSummary file /girder/girder/api/rest.py, line 630 in endpointDecorator>", "<FrameSummary file /girder/girder/api/rest.py, line 1230 in POST>", "<FrameSummary file /girder/girder/api/rest.py, line 970 in handleRoute>", "<FrameSummary file /girder/girder/api/access.py, line 63 in wrapped>", "<FrameSummary file /girder/girder/api/describe.py, line 709 in wrapped>", "<FrameSummary file /girder/plugins/wholetale/server/rest/account.py, line 339 in addAccountKey>", "<FrameSummary file /girder/plugins/wholetale/server/lib/dataverse/auth.py, line 16 in verify>", "<FrameSummary file /girder/venv/lib/python3.9/site-packages/requests/api.py, line 76 in get>", "<FrameSummary file /girder/venv/lib/python3.9/site-packages/requests/api.py, line 61 in request>", "<FrameSummary file /girder/venv/lib/python3.9/site-packages/requests/sessions.py, line 542 in request>", "<FrameSummary file /girder/venv/lib/python3.9/site-packages/requests/sessions.py, line 655 in send>", "<FrameSummary file /girder/venv/lib/python3.9/site-packages/requests/adapters.py, line 516 in send>"], "type": "internal"}

• Connect to DataONE

1. Click "Connect Account" under the DataONE repository
2. Confirm that you're redirected to the DataONE login
3. Log in with ORCID
4. Confirm that you're redirected back to Whole Tale
5. Confirm that you see "Authorized" under the DataONE server
6. Confirm that the "Connect Account" Button is disabled
7. Disconnect your account
8. Confirm that you can re-connect it

• Confirm tokens retained across logins
  #370

1. Connect one or more accounts as outlined above
2. Logout and login again
3. Using Swagger UI, hit the /user/me endpoint to confirm tokens are still present

Tale Creation

• Create RStudio Tale

1. Select the Create New Tale button on the 'Browse' page
2. Enter title "Test Tale"
3. Select on of the RStudio instances
4. Select "Create New Tale"
5. Confirm that you're brought to the 'Run' page and that the Tale isn't building
6. Click 'Run Tale'
7. Confirm notification dialog opens
8. Confirm that the build/launch steps are correct
9. Wait for the building to complete
10. Navigate to the Browse page
11. Confirm the Tale is shown under "Currently running"

TEST: There are two different RStudio images, so one is missing a test case
Tale failed to Build the first time, but worked when trying again

#4 sha256:c797c633effb523ebfce4b651d3e8cbdc155fdfee8bdc7f1658d9be0a63dca97 238.87MB / 444.78MB 23.3sHTTPError: 502 Server Error: Bad Gateway for url: https://girder.stage.wholetale.org/api/v1/job/6089dfee22bad715637ae558
  File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 412, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/girder_worker/task.py", line 148, in __call__
    results = super(Task, self).__call__(*_t_args, **_t_kwargs)
  File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 704, in __protected_call__
    return self.run(*args, **kwargs)
  File "/gwvolman/gwvolman/tasks.py", line 433, in build_tale_image
    ret = _build_image(
  File "/gwvolman/gwvolman/utils.py", line 356, in _build_image
    print(line.decode('utf-8').strip())
  File "/usr/local/lib/python3.8/dist-packages/girder_worker/utils.py", line 158, in write
    self._write_func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/girder_worker/utils.py", line 259, in write
    self._flush()
  File "/usr/local/lib/python3.8/dist-packages/girder_worker/utils.py", line 237, in _flush
    req.raise_for_status()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)

• Compose Jupyter Notebook Tale

1. TODO: Needs test case

This is my default env

• Compose JupyterLab Tale

1. TODO: Needs test case

Tale built and ran successfully

• Compose JupyterLab with Spark

1. TODO: Needs test case

It builds, I guess? I don't know anything about Spark, so not sure how to test this

• Compose MATLAB Desktop Tale

1. Create tale from Git repo https://github.com/whole-tale/matlab-example/
2. Run multiplicative_arima_example.m, confirm outputs
3. Confirm CSP settings

TEST: Are these the Content Security Policy settings in the browser? How do I check these? Need more steps here
FAIL: MATLAB could not run the script:

• Compose MATLAB Jupyter Kernel Tale

1. Create tale from Git repo https://github.com/whole-tale/matlab-example/
2. Run multiplicative_arima_example.ipynb, confirm outputs match multiplicative_arima_example.html

• Compose MATLAB Linux Desktop Tale

1. Create tale from Git repo https://github.com/whole-tale/matlab-example/
2. Run multiplicative_arima_example.m, confirm outputs

TEST: Duplicated test case
FAIL: MATLAB could not run the script:

• Compose STATA Desktop Tale

1. Create tale from Git repo https://github.com/whole-tale/stata-example/
2. Run example.do, confirm outputs

• Compose STATA Jupyter Kernel Tale

1. Create tale from Git repo https://github.com/whole-tale/stata-example/
2. Run example.ipynb, confirm outputs match example.html

FAIL: %help summarize hangs, everything else up to that and after appear to work properly
WARN: warning appears in example notebook towards the end

NOTE: A newer version of stata_kernel exists. Run

	pip install stata_kernel --upgrade

to install the latest version.

• Compose OpenRefine Tale

1. TODO: Needs test case

WARN: Newer version available
It builds and runs, I guess? I don't know anything about OpenRefine, so not sure how else to test this.. additional steps needed

• Too many instances

1. Make sure you have two launched Tales
2. Attempt to create a Tale
3. Confirm that you receive an error that you have the max limit
4. Confirm that the Tale is created, but not the instance

Analyze in Whole Tale

Task 1: Importing a Dataset from Dataverse

These test cases cover potential situations that can occur when importing datasets from Dataverse.

• Case 1: Ideal Scenario:

Testing Steps:

1. Remove all running Tale instances
2. Navigate to https://girder.stage.wholetale.org/api/v1/integration/dataverse?datasetPid=doi%3A10.7910%2FDVN%2F3MJ7IR&siteUrl=https%3A%2F%2Fdataverse.harvard.edu
3. Confirm that the Tale title reads as Replication Data for: "Agricultural Fires and Health at Birth"
4. Confirm that the only item in the Input data section matches the uri with Data Source appended
5. Confirm that no environment is selected
6. Confirm that the Create New Tale button is disabled
7. Select an environment
8. Confirm that READ/WRITE is selected
9. Click Create New Tale
10. Confirm notification progress and success
11. Confirm that you are redirected to the run page
12. Confirm that the Tale name matches the Tale Name in the Create Tale Modal
13. Confirm that the data exists under Tale Workspace
14. Confirm that the Tale's category is science

FAIL: Not automatically redirected
FAIL: Tale creation/import hangs at "Initializing..."
FAIL: Tale is automatically launched with AiWT cases (is this expected?)

Task 2: Importing a Dataset from DataONE

These test cases cover different scenarios that may happen when importing datasets from DataONE.

• Case 1: Ideal Scenario:

This test covers the ideal case where dataset importing has all of the needed information
to properly run. This includes the dataset location, title, and selected environment.

Testing Steps:

1. Remove all running Tale instances
2. Navigate to https://girder.stage.wholetale.org/api/v1/integration/dataone?uri=https%3A%2F%2Fsearch.dataone.org%2Fview%2Fdoi%3A10.18739%2FA2VQ2S94D&title=Fire%20influences%20on%20forest%20recovery%20and%20associated%20climate%20feedbacks%20in%20Siberian%20Larch%20Forests%2C%20Russia&environment=RStudio
3. Confirm that the Tale title reads as Fire influences on forest recovery and associated climate feedbacks in Siberian Larch Forests, Russia
4. Confirm that the only item in the Selected data section matches the uri with Data Source appended
5. Confirm that Rstudio is selected in the Environments widget
6. Confirm that the Create New Tale button is enabled
7. Confirm READ ONLY is selected
8. Click Create New Tale
9. Confirm that you are redirected to the run page
10. Confirm that the Tale name matches the Tale Name in the Create Tale Modal
11. Confirm that the data exists in the Tale under External Data
12. Confirm that the Tale's category is science

FAIL: Not automatically redirected
FAIL: Strange delay before Tale goes into Initializing...
FAIL: Tale creation/import hangs at "Initializing..."
WARN: Tale is automatically launched with AiWT cases (is this expected?)

• Case 2: Testing Different DataONE Deployments
  The Analyze in Whole Tale feature is deployed across a few DataONE test sites. This test ensures Whole Tale has the ability to import
  data from each of them.

Links:

1. https://dashboard.stage.wholetale.org/browse?api=https%3A%2F%2Fcn-stage-2.test.dataone.org%2Fcn%2Fv2&name=Dataone%20Dataset&uri=https://dev.nceas.ucsb.edu/view/doi:10.5072/FK2K075M25
2. https://dashboard.stage.wholetale.org/browse?api=http%3A%2F%2Fcn-stage.test.dataone.org%2Fcn%2Fv2&name=Dataone%20Dataset&uri=https://test.arcticdata.io/view/doi:10.5065/D6N014NC

Testing Steps:

1. Remove all running Tale instances
2. Navigate to each of the two links above
3. Select an environment
4. Click Create New Tale
5. Confirm that the correct data exists in the Tale under External Data

WARN: No convenient way (via the UI) to abort while "starting" a Tale
FAIL: AiWT cases may no longer respect the instance limit:

Task 3: Shared Behavior

These cases show errors that are common between Dataverse and DataONE. Although they are tested using DataONE URIs, the errors should be the same when replacing it with a Dataverse URI.

• Case 1: Missing Dataset Title Query Param:

This test covers the case where the name query parameter is missing. The expected behavior is
that the uri query parameter is used instead.

Testing Steps:

1. Remove all running Tale instances
2. Navigate to https://dashboard.stage.wholetale.org/browse?uri=https%3A%2F%2Fdataverse.harvard.edu%2Fapi%2Faccess%2Fdatafile%2F3323458
3. Confirm that the Tale title reads as https://dataverse.harvard.edu/api/access/datafile/3323458
4. Confirm that the only item in the Selected data section matches the uri with Data Source appended
5. Confirm that no Environment is selected
6. Confirm that the Create New Tale button is disabled
7. Select an environment
8. Click Create New Tale
9. Confirm that the notification bar appears & properly updates
10. Confirm that you are redirected to the run page
11. Confirm that the Tale name matches the Tale Name in the Create Tale Modal
12. Confirm that the data exists in the Tale under External Data

FAIL: Tale name not defaulted to uri

• Case 2: Importing Data with Read/Write:

This test covers the case where a user imports data from an external source and selects READ/WRITE in the Create Tale Modal.

Testing Steps (Variant 1):

1. Remove all running Tale instances
2. Navigate to https://dashboard.stage.wholetale.org/browse?uri=https%3A%2F%2Fdataverse.harvard.edu%2Fapi%2Faccess%2Fdatafile%2F3323458
3. Confirm that the Tale title matches the URI above
4. Confirm that the only item in the Selected data section matches the uri with Data Source appended
5. Confirm that the Create New Tale button is disabled
6. Select an environment
7. Select READ/WRITE
8. Click Create New Tale
9. Confirm that the notification bar appears & properly updates
10. Confirm that you are redirected to the run page
11. Confirm that the Tale name matches the Tale Name in the Create Tale Modal
12. Confirm that the data exists in the Tale Workspace

FAIL: Not redirected to run page

Testing Steps (Variant 2):

1. Remove all running Tale instances
2. Navigate to https://dashboard.stage.wholetale.org/browse?api=https%3A%2F%2Fdev.nceas.ucsb.edu%2Fknb%2Fd1%2Fmn%2Fv2&environment=JupyterLab&name=proveit&uri=doi%3A10.5072%2FFK27P92Z55
3. Confirm that the Tale title matches the dataset
4. Confirm that the only item in the Selected data section matches the uri with Data Source appended
5. Confirm that the Create New Tale button is disabled
6. Select READ/WRITE
7. Click Create New Tale
8. Confirm that the notification bar appears & properly updates
9. Confirm that you are redirected to the run page
10. Confirm that the Tale name matches the Tale Name in the Create Tale Modal
11. Confirm that the data exists in the Tale Workspace

Tale metadata tests

The purpose of these tests are to confirm that the metadata files (manifest.json, environment.json, LICENSE) we generate are correct.

• manifest.json

1. Launch a Tale that you own
2. Add a dataset to the Tale
3. Add at least one author
4. Add at least one file to the workspace
5. Export the Tale (as BagIt)
6. Open the manifest.json file under metadata/
7. Confirm that the author is listed under schema:author
8. Confirm that you are listed under createdBy
9. Confirm that the file(s) in the workspace are properly listed in aggregates
10. Confirm that the external dataset is referenced in aggregates
11. Confirm that the dataset is also listed under Datasets

FAIL: No "Datasets" section (but there is "wt:usesDataset"? are these the same?)

• environment.json

1. Open the archive from the manifest.json test
2. Navigate to metadata/environment.json
3. Open the girder web API
4. Locate and GET the Image that the Tale used
5. Confirm that the config section from girder matches the config section in environment.json

• LICENSE

1. Open the archive from the manifest.json test
2. Open the LICENSE file
3. Navigate to Run > metadata
4. Confirm that the opened LICENSE file matches the selected license on the metadata page
5. Change the Tale's license
6. Export the Tale
7. Confirm that the LICENSE changed appropriately

FAIL: LICENSE file in exported Tale always has CC 4.0 text, even after changing, saving, and re-exporting

Tale Sharing Tests

These tests are centered around the feature for sharing Tales with other Whole Tale
users. To run these tests, register at least one other account, referred to as User B.

• Case 1: Sharing with 'Can View' permissions

  1. Create a new Tale
  2. Add data to the workspace and add at least one dataset
  3. Navigate to the 'Share' tab
  4. Share the Tale with User B
  5. Confirm that the default permission is 'Can View'
  6. Switch the account to User B
  7. Navigate to the 'Shared with Me' tab on the Browse page
  8. Confirm that you see the Tale from step 1
  9. Confirm that you can't edit the Workspace, External Data, or metadata

• Case 2: Sharing with 'Can Edit' permissions

  1. Create a new Tale
  2. Add data to the workspace and add at least one dataset
  3. Navigate to the 'Share' tab
  4. Share the Tale with User B
  5. Set the permissions to 'Can Edit'
  6. Switch the account to User B
  7. Navigate to the 'Shared with Me' tab on the Browse page
  8. Confirm that you see the Tale from step 1
  9. Remove one of the files from the Workspace
  10. Add a dataset to External Data
  11. Change a field in the metadata (title, description, etc)
  12. Switch to User A
  13. Confirm that the changes made to the Tale persisted

Publishing tests

• Case 1: No Accounts Configured

  1. Disconnect all of your external accounts
  2. Launch a Tale that you own
  3. Click the ellipsis dropdown on the Run page
  4. Click Publish tale...
  5. Confirm that your are notified to link accounts on the settings page
  6. Click Go to Settings
  7. Confirm you are re-directed to settings

• Case 2: Publishing to a Third Party

  1. Connect to a third party provider (make sure it's a dev server)

  2. Launch a Tale that you own

  3. Add some data to it

  4. Click Publish tale...

  5. Confirm that the publishing modal opens

  6. Confirm that the third party that you are connected to appears in the dropdown

  7. Select the third party

  8. Click Publish

  9. Once complete, navigate to the metadata page

  10. Confirm that the published location is present

  11. Visit the published location

  12. Confirm that you see the following files plus yours

      1. metadata.xml
      2. LICENSE
      3. README.md
      4. manifest.json
      5. environment.json

  13. Repeat for each third party provider

      a. After publishing to DataONE, use girder to query the Tale

      b. Ensure that the 'pid' field in the 'publishInfo' object has an identifier in a doi format

Zenodo: PASS
DataONE: FAIL - "Not enough segments" error returned when publishing.. re-authed with DataONE to refresh my token, but same behavior. Cannot publish to DataONE.

• Case 3: Re-published Tale

1. Launch a Tale that has been published before
2. Take note of the Published Location in Run > metadata
3. Re-publish the Tale to the third party
4. Confirm that the Published Location has changed to the correct package landing page

Zenodo integration tests

• Register Zenodo data

1. Select Manage > Data tab
2. Select "+". Confirm "Search for Data Sources" modal displays
3. Search for Concept DOI doi:10.5281/zenodo.16384
4. Confirm "Belmont Forum..." dataset is found on Zenodo with suffix ver_1
5. Search for URL https://zenodo.org/record/1172960#.XidIvlNKh24
6. Confirm "Belmont Forum..." dataset is found on Zenodo with suffix ver_2
7. Search for DOI doi:10.5281/zenodo.1172960
8. Confirm "Belmont Forum..." dataset is found on Zenodo with suffix ver_2
9. Search for Concept DOI doi:10.5281/zenodo.608984
10. Confirm "Belmont Forum..." dataset is found on Zenodo with suffix ver_2
11. Select Register
12. Confirm dataset appears in Manage Data list
13. Add to tale and confirm contents match

The register tests the following cases.

• Datasets with multiple files
• Datasets with multiple versions
• Concept DOI

TEST: There is no Manage > Data tab
TEST: Cannot browse contents of External Data via the UI, not sure how to verify contents

• Zenodo dataset via AiWT

1. Open https://dashboard.stage.wholetale.org/browse?uri=https%3A%2F%2Fdoi.org%2F10.5281%2Fzenodo.820575&name=Automotive%20Sensor%20Data
2. Confirm Source Data URL and Title match above URL and "READ ONLY" is selected
3. Select environment
4. Select 'Create New Tale'
5. Confirm dataset is mounted read-only, citation is as expected, zipfile is not extracted

The register tests the following cases.

• Dataset with single file (zipfile)

FAIL: No /data (or /home or /workspace) mounted to notebook, cannot run notebook

• Zenodo Binder via AiWT

1. Open https://dashboard.stage.wholetale.org/browse?uri=https%3A%2F%2Fdoi.org%2F10.5281%2Fzenodo.3242073&environment=JupyterLab&name=Simple%20requirements.txt%20based%20example%20for%20repo2docker&asTale=true
2. COnfirm Source Data and Title match URL and "READ WRITE" is selected
3. Select 'Create Tale'
4. Confirm image builds and notebook can run

FAIL: Image failed to build, cannot view build logs {"message": "Read access denied for job 608b1fc622bad715637af6e5 (user 608b1a8f22bad715637af658).", "type": "access"}
(iirc, this also happened during review, but we couldn't figure out what had happened)

• Zenodo Tale via AiWT

1. Open https://girder.stage.wholetale.org/api/v1/integration/zenodo?doi=10.5072%2Fzenodo.490450&resource_server=sandbox.zenodo.org
2. Confirm Water Tale is created.
3. Confirm citation on Metadata view
4. Confirm DOI matches above
5. Use the Swagger UI to confirm the publishInfo field on the tale object contains pid, repository, repository_id, uri
6. Run the tale and and notebook, confirm it works as expected (produces glorious map of Texas)

• Publish/import to/from Zenodo Sandbox

1. Follow instructions above to configure your API key for Zenodo sandbox
2. Create your own Tale and publish to sandbox.zenodo.org
3. Confirm publishInfo contains Sandbox entry
4. Delete tale
5. Import tale from Zenodo
6. Confirm publishInfo
7. Modify the tale
8. Re-publish to sandbox.zenodo.org
9. Delete tale
10. Import tale from Zenodo
11. Confirm publishInfo
12. Download and extract the exported Tale
13. Execute the run-local.sh command to run the Tale locally

FAIL: Publish works, import hangs on "Copying files to workspace" (noting that the workspace in my case is completely empty: https://sandbox.zenodo.org/record/804485)

Versioning Tests

• Tale Versioning

1. View a Tale that you own
2. Expand Tale Version panel (on the right)
3. Create a New Version: init
4. Modify metadata, upload a file, and attach a dataset to the Tale
5. Create another new version: modified
6. While viewing Run > Files > External Data, restore from init - you should see no datasets
7. Rename the modified Version to something else
8. In the same view, restore back to something else - you should see the dataset you added
9. Run the tale. Confirm the versions are available under ../versions and that the contents including metadata are as expected
10. View Info for a version
11. Rename a version. Confirm rename in container.
12. Export a version. Confirm exported version matches selected version.
13. Delete a version. Confirm deletion in container.

FAIL: Datasets / Metadata changes are not preserved when renaming a version. Oddly, Tale Workspace was preserved correctly.

• Copy versions

1. Login as a user with view-only permissions of a tale with versions.
2. Run the tale and confirm versions are copied

FAIL: Versions are copied, but "created" timestamps are not preserved

• Publish/export version

1. Create a new tale
2. Upload files but do not create a version
3. Publish the tale
4. Note that there is a new version in the version panel
5. Confirm that the data files in the published artifact are correct
6. Repeat for export
7. Confirm manifest contents including vocabulary

Git Integration Tests

• Compose Tale from Git Repository

1. Select the Create New Tale button on the 'Browse' page
2. Select the Create Tale from Git Repository option
3. Paste a link to a Git repository in the correct field
4. Add a name and pick a compute environment
5. Create the Tale
6. Confirm notification progress and success
7. Confirm that the git repository was cloned to the workspace/ root
8. Run the tale
9. Confirm interaction via git tools (may require environment customization)

Overlaps with "Compose XX Tale" Tale tests, whose Git capabilities functioned as expected 👍

• Connect to Git

1. Create a new empty tale
2. Select "Connect to Git Repository" menu item from Tale menu
3. Confirm notification progress and success
4. Once complete, navigate to the Tale workspace and confirm that the repository has been cloned

WARN: Log output says "Tale created" - this is likely a copy paste error

Other tests

• Citations

1. Add files from multiple external datasets (e.g.: doi:10.5281/zenodo.3381658, doi:10.5065/D6862DM8, doi:10.5063/F10K26VR)
2. Confirm Datasets Used citations and related identifiers are updated

Event Notification Tests

• Tale sharing/unsharing

1. Login as user1 and user2 in separate browser sessions
2. As user2, select "Shared with me"
3. As user1, share a tale with user2; as user2 confirm the tale appears
4. As user2, select "View" on the shared tale
5. As user1 and 2 edit the tale metadata
6. As user1 save changes; as user2 confirm Tale edited modal
7. As user1, unshare the tale; as user2 confirm Tale unshared modal
8. As user1, re-share with user2
9. As user2, select "Shared with Me"
10. As user1, delete the shared tale; as user2 confirm that the tale disappears

• Import tale

1. Import a tale from Github, Zenodo, and as Binder
2. Confirm metadata updates accordingly after import completes

FAIL: See AiWT Test Cases - too many import cases broken, and I suspect that those failing cases will affect this one

Vocabulary Tests

• Vocabularies

1. Go to https://vocabularies.stage.wholetale.org/wt/1.0/index.html
2. Confirm that it matches https://whole-tale.github.io/serialization-format/wt/1.0/

FAIL: https://vocabularies.stage.wholetale.org/wt/1.0/index.html only returns 404 page not found

Forward Authentication Test

• Forward authentication

1. Start a tale
2. Confirm that the tale is accessible via iframe and popout
3. Open an incognito window and access the tale URL
4. Confirm that you are prompted for authentication and can access the tale once authenciated
5. Login as a user who does not have permission to access the tale
6. Confirm that they receive an access denied errror

FAIL: I am correctly prompted for login when accessing (for example) https://tmp-zf2ypkftc9yu.stage.wholetale.org/login?next=%2Ftree, but after authing I see this view indicating that I require access to the container logs to continue: