Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dcache or samba catalog item #169

Open
wants to merge 134 commits into
base: main
Choose a base branch
from
Open

Dcache or samba catalog item #169

wants to merge 134 commits into from

Conversation

sverhoeven
Copy link
Member

@sverhoeven sverhoeven commented Sep 11, 2024 •
edited
Loading

Fixes #166

TODO

  • merge main and grader
  • merge grader-samba, leaving dcache stuff intact
  • name for variable that is either dcache or samba: some suggestion data_lake_format, readonly_data_type, storage_type, data_source, data_mount_type, shared_data_source
  • add shared_data_source variable to pick either samba or dcache (or none??)
  • remove worker/jumphost cluster stuff, as its no longer needed as we will have single file server and a multiple jupyter hub machines started as separate workspaces.
  • Move dcache macaroon to org Store macaroon in Collaborative organisations secret #168
  • test with vagrant
    • dcache
    • samba
  • create component and catalog item on SRC
  • test on SRC
    • dcache without students
    • dcache with students
    • samba without students
    • samba with students
  • Remove SRC component and catalog items that are not used
  • Add deprecated to name of SRC component and catalog items that are used
  • just before merge rename occurrences of dcache-or-samba to main in repo and in SRC.
  • after merge mail Rolf that he should use eWaterCycle dcache or eWaterCycle samba + Samba server catalog items.

To test

  1. Follow instructions to create a workspace, either
    2. making the component/catalog item your self or
    3. use eWaterCycle dcache and eWaterCycle samba catalog items I made.

In ewatercycle-nlesc collab org I already created required secrets and private network.

sverhoeven and others added 30 commits February 13, 2024 13:02
@sverhoeven
Setup nbgrader in one class, one grader mode
377783d
@sverhoeven
Add student creation via interacve src parameter + try e2xgrader
605ff3a
@sverhoeven
Drop e2xgrader is not a lab extension yet, removing it
77a8cd2
@sverhoeven
Improve docs
214f652
@sverhoeven
Add sudo
aa9c7ef
@sverhoeven
No longer need experimental env var with latest vagrant
52368c7
@sverhoeven
Fix license
c55b2fe
@sverhoeven
Fix ERROR: Failed to find specified config file: /etc/jupyterhub/jupy…
ab489be 
…terhub_config.py in `systemctl status jupyterhub`
@sverhoeven
Add course repo + refactor + fix lint errors
648ad57
@sverhoeven
Dont use deprecated include
6d1b87d 
ERROR! [DEPRECATED]: ansible.builtin.include has been removed. Use include_tasks or import_tasks instead. This feature was removed from ansible-core in a release after 2023-05-16. Please update your playbooks.
Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.
@sverhoeven
nbgrader does subprocess alembic, but was not in path so activate con…
1195457 
…da env first
@sverhoeven
Set PATH env var so alembic can be found
ee6ab9a
@sverhoeven
Update TEACH.md
37e6ab5 
Use value from defaults
@sverhoeven
Test out named volumes + cluster
1d35fb2
@sverhoeven
Merge branch 'grader' of github.com:eWaterCycle/infra into grader
5b78a67
@sverhoeven
Mirror rs dat more
178632d
@sverhoeven
Expose worker_ip_addresses as plugin resource parameter
eb66ff2
@sverhoeven
unneeded if
326a8a2
@sverhoeven
Skip cluster
58a91ac 
worker_ip_addresses is not passed to https://gitlab.com/rsc-surf-nl/plugins/plugin-external-plugin/-/blob/main/plugin-external-plugin.yml?ref_type=heads#L156
@sverhoeven
Really skip cluster, left is named storage
01a0a8c
@sverhoeven
Expose attached storage items as interactive parameters
1938143
@sverhoeven
re-enable all roles
dbec610
@sverhoeven
Better docs
dc5df4e
@sverhoeven
Merge TEACH.md into other docs
5575997
@sverhoeven
Pick right branch
ce4c108
@sverhoeven
Add all non grader non system users as students + disable some nbgrad…
fbe8706 
…er extennsions based on being grader_user
@sverhoeven
Dump worker_ip_addresses
ac455f0
@sverhoeven
false is not a command
1e50645
@sverhoeven
List users in /home
1534fec
@sverhoeven
More todos
012f1ac
sverhoeven and others added 27 commits September 17, 2024 12:45
@sverhoeven
Make samba mount work with vagrant
2ff26fd
@sverhoeven
Document vars for vagrant file server + update populate data chapters
a3e85b5
@sverhoeven
Make readme focus on workspace creation + moved other chapters to own…
bcb8876 
… markdown files
@sverhoeven
Renest
5d3e800
@sverhoeven
Skip parameter sets when shared data source is samba
4d04d4f
@sverhoeven
Pin to latest ewatercycle
cbbafcc
@sverhoeven
Ignore just space in students
ba002bb 
As SRC does not allow empty value in form
@sverhoeven
Created component and dcache catalog item
312e9fd
@sverhoeven
Clone dcache catalog item to samba catalog item
96104df
@sverhoeven
Update workspace parameters screenshot
2d33ca8
@sverhoeven
correct version
7f4bdb8
@sverhoeven
Handle ' ' as students value
a297bd2
@sverhoeven
Mark students as optional
04275fc
@sverhoeven
Dont use ==USERNAME==, does not work for parameters
034b0d8
@sverhoeven
nbgrader labextensions where moved to @jupyter namespace
481b25d
@sverhoeven
Dont create students posix accounts when students is undefined
59ecbd3
@sverhoeven
Make sure passlib is installed for vagrant up
dea68eb
@sverhoeven
Trying to spin up src workspace with samba as data source
0700062
@sverhoeven
Merge remote-tracking branch 'origin/dcache-or-samba' into dcache-or-…
0043b38 
…samba
@sverhoeven
Add AI disclaimer according with https://doi.org/10.5281/zenodo.10363728
bf6ed1c
@sverhoeven
List paths that ewatecycle package and esmvaltool expect
745264e
@sverhoeven
Make ERA5 download optional
c527574
@sverhoeven
Add grdc dir so import ewatercycle does not complain
7395fb0
@sverhoeven
Run playbook with screen
91f4cff
@sverhoeven
Skip ERA5 download when cds vars are None
dd374f0
@sverhoeven
Add missing var
41d3b1b
@sverhoeven
No need to mkdir grdc already done in grdc role
341dd3c
@sverhoeven sverhoeven marked this pull request as ready for review October 14, 2024 11:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@BSchilperoort BSchilperoort Awaiting requested review from BSchilperoort

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

one catalog item
2 participants
@sverhoeven @BSchilperoort