From a6e91e48c171a59820ced6fe1054b7b44789114f Mon Sep 17 00:00:00 2001 From: Alex Domingo Date: Tue, 16 May 2023 18:17:21 +0200 Subject: [PATCH 01/10] add instructions on how to check quota information in VSC account page --- source/access/managing_disk_usage.rst | 35 +++++++++++++++++++++------ 1 file changed, 28 insertions(+), 7 deletions(-) diff --git a/source/access/managing_disk_usage.rst b/source/access/managing_disk_usage.rst index 08a08fb0..76ff9306 100644 --- a/source/access/managing_disk_usage.rst +++ b/source/access/managing_disk_usage.rst @@ -1,22 +1,43 @@ .. _disk usage: +############################### How much disk space am I using? -=============================== +############################### .. _quota: Total disk space used on file systems with quota ------------------------------------------------- +================================================ On file systems with quota enabled, you can check the amount of disk space that is available for you, and the amount of disk space that is in use by you. -On most systems, ``myquota`` will show you for the ``$VSC_HOME``, -``$VSC_DATA`` and ``$VSC_SCRATCH`` file systems either the -percentage of the available disk space you are using, or the -absolute amount. Users from Ghent university should check their disk usage -using the `web application `_. +VSC account page +---------------- + +Your VSC account page shows up-to-date information about your storage usage on +each of the file systems accessible to your account. This information is +available in the *Usage* section of the +`View Account `_ tab and includes: + +* total data size +* current quota limit +* percentage use +* number of files used (*inodes*) + +You will find the usage data for your :ref:`personal storage ` +space such as ``VSC_HOME``, ``VSC_DATA`` and ``VSC_SCRATCH`` as well as your +:ref:`Virtual Organization ` if you are in one. + +Terminal in the cluster +----------------------- + +On the system of most VSC clusters, ``myquota`` will show you for the +``$VSC_HOME``, ``$VSC_DATA`` and ``$VSC_SCRATCH`` file systems either the +percentage of the available disk space you are using, or the absolute amount. +Users from Ghent university should check their disk usage using the `web +application `_. If quota have been set on the number of files you can create on a file system, those are listed as well. From c5c0775c81d400d8b25ea937a5654c10ec2e7c2b Mon Sep 17 00:00:00 2001 From: Alex Domingo Date: Tue, 16 May 2023 18:17:51 +0200 Subject: [PATCH 02/10] add link to how to check quota in data storage FAQ --- source/access/where_can_i_store_what_kind_of_data.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/source/access/where_can_i_store_what_kind_of_data.rst b/source/access/where_can_i_store_what_kind_of_data.rst index 0593a48d..b5b5bdfd 100644 --- a/source/access/where_can_i_store_what_kind_of_data.rst +++ b/source/access/where_can_i_store_what_kind_of_data.rst @@ -61,6 +61,10 @@ open for the general quota warnings! The same holds for running jobs that need to write files: when you reach your hard quota, jobs will crash. +If you reached the limit in any of your quotas, it might be possible to +increase them if your need more space to carry out your research projects. +Check :ref:`more quota` for more information. + A few additional site-specific remarks concerning data storage can be found on the following pages: From 29f5ba1d85d698d6b7d8bc320ee09231ee6e4e70 Mon Sep 17 00:00:00 2001 From: Alex Domingo Date: Tue, 16 May 2023 18:18:06 +0200 Subject: [PATCH 03/10] add FAQ about how to request an increase of quota --- source/access/how_to_request_more_quota.rst | 89 +++++++++++++++++++++ source/faq.rst | 5 +- 2 files changed, 92 insertions(+), 2 deletions(-) create mode 100644 source/access/how_to_request_more_quota.rst diff --git a/source/access/how_to_request_more_quota.rst b/source/access/how_to_request_more_quota.rst new file mode 100644 index 00000000..830174d5 --- /dev/null +++ b/source/access/how_to_request_more_quota.rst @@ -0,0 +1,89 @@ +.. _more quota: + +################################## +How can I request more disk quota? +################################## + +If the current quota limits of your :ref:`personal storage ` or +:ref:`Virtual Organization (VO) ` are not large enough to +carry out your research project, it might be possible to increase them. This +option depends on data storage policies of the site managing your VSC account, +VO or Tier-1 project as well as on current capacity of the storage system. + +Before requesting more storage, please check carefully the +:ref:`current data usage of your VSC account ` and identify which +file system needs a larger quota. + +Personal storage +================ + +The following are general guidelines to request more storage in the personal +file systems of your VSC account: + +VSC_HOME + The storage of your home directory is small on purpose. If you need more + storage, please switch to ``VSC_DATA``. + +VSC_DATA + On VSC sites that support :ref:`Virtual Organizations (VO) + `, it is preferable to join a VO to get access to a + larger ``VSC_DATA_VO``. + If that is not the case or if in doubt, please contact :ref:`user support + `. + +VSC_SCRATCH + On VSC sites that support :ref:`Virtual Organizations (VO) + `, it is preferable to join a VO to get access to a + larger ``VSC_SCRATCH_VO``. + If that is not the case or if in doubt, please contact :ref:`user support + `. + +Virtual Organizations +===================== + +VSC_DATA_VO, VSC_SCRATCH_VO + The quota of the data storage in your :ref:`Virtual Organization (VO) + ` is managed by the moderator of the VO, who is typically + somebody in your research group. The moderator can manage all quotas of the VO + in the `Edit VO `_ tab of the VSC + account page. + +Requesting more storage space +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +VO moderators can request additional quota for ``VSC_DATA_VO`` and ``VSC_SCRATCH_VO``: + +#. Go to the section **Request additional quota** in the + `Edit VO `_ page + +#. Fill in the amount of additional storage you want for ``VSC_DATA_VO`` + (labelled ``VSC_DATA`` in this section) and/or ``VSC_SCRATCH_VO`` (labelled + ``VSC_SCRATCH_XXX`` in this section) + +#. Add a comment explaining why you need additional storage space + +#. Submit the request by clicking on the *Submit request* button + +#. Your request will be reviewed by the HPC administrators + +Setting per-member VO quota +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +VO moderators can tweak the share of the VO quota that each member can +maximally use. By default, this is set to 50% of the total quota for each user. + +#. Go to the section **Request additional quota** in the + `Edit VO `_ page + +#. Adjust the share (%) of the available space available to each user + +#. Submit the request by clicking on the *Confirm* button + +#. The per-member VO quota will be updated in 30 minutes maximum + +.. note:: + + The sum of all user quotas in your VO can be above 100%. The share + for any user indicates what he/she can maximally use, but the actual limit + will then depend on the usage of the other members. The total storage space + of the VO will always be respected. diff --git a/source/faq.rst b/source/faq.rst index 45191ab1..a9b89ac3 100644 --- a/source/faq.rst +++ b/source/faq.rst @@ -13,7 +13,7 @@ General questions How can I contact user support? how_do_i_acknowledge_the_vsc_in_publications - what_are_standard_terms_used_in_hpc.rst + what_are_standard_terms_used_in_hpc .. _access faqs: @@ -26,7 +26,8 @@ Access to the infrastructure I messed up my authentication keys, what can I do? access/where_can_i_store_what_kind_of_data - access/managing_disk_usage.rst + access/managing_disk_usage + access/how_to_request_more_quota .. _job faqs: From a50e51a3d6070d40e214fed1f6fadab1c6bb9dbf Mon Sep 17 00:00:00 2001 From: Alex Domingo Date: Wed, 17 May 2023 15:11:49 +0200 Subject: [PATCH 04/10] clarify who is usually moderator in a VO Co-authored-by: Sam Moors --- source/access/how_to_request_more_quota.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/access/how_to_request_more_quota.rst b/source/access/how_to_request_more_quota.rst index 830174d5..1b56bb15 100644 --- a/source/access/how_to_request_more_quota.rst +++ b/source/access/how_to_request_more_quota.rst @@ -44,7 +44,7 @@ Virtual Organizations VSC_DATA_VO, VSC_SCRATCH_VO The quota of the data storage in your :ref:`Virtual Organization (VO) ` is managed by the moderator of the VO, who is typically - somebody in your research group. The moderator can manage all quotas of the VO + the leader of your research group. The moderator can manage all quotas of the VO in the `Edit VO `_ tab of the VSC account page. From c3133db000e5fa40a462a85b556853316316d1e5 Mon Sep 17 00:00:00 2001 From: Alex Domingo Date: Wed, 17 May 2023 15:12:26 +0200 Subject: [PATCH 05/10] Edit VO is shown as a tab Co-authored-by: Sam Moors --- source/access/how_to_request_more_quota.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/access/how_to_request_more_quota.rst b/source/access/how_to_request_more_quota.rst index 1b56bb15..b91813c2 100644 --- a/source/access/how_to_request_more_quota.rst +++ b/source/access/how_to_request_more_quota.rst @@ -54,7 +54,7 @@ Requesting more storage space VO moderators can request additional quota for ``VSC_DATA_VO`` and ``VSC_SCRATCH_VO``: #. Go to the section **Request additional quota** in the - `Edit VO `_ page + `Edit VO `_ tab #. Fill in the amount of additional storage you want for ``VSC_DATA_VO`` (labelled ``VSC_DATA`` in this section) and/or ``VSC_SCRATCH_VO`` (labelled From c2c2208d750a23d21e647d82518cdb9389339629 Mon Sep 17 00:00:00 2001 From: Alex Domingo Date: Wed, 17 May 2023 15:12:34 +0200 Subject: [PATCH 06/10] Edit VO is shown as a tab Co-authored-by: Sam Moors --- source/access/how_to_request_more_quota.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/access/how_to_request_more_quota.rst b/source/access/how_to_request_more_quota.rst index b91813c2..d7ac30a3 100644 --- a/source/access/how_to_request_more_quota.rst +++ b/source/access/how_to_request_more_quota.rst @@ -73,7 +73,7 @@ VO moderators can tweak the share of the VO quota that each member can maximally use. By default, this is set to 50% of the total quota for each user. #. Go to the section **Request additional quota** in the - `Edit VO `_ page + `Edit VO `_ tab #. Adjust the share (%) of the available space available to each user From 68496a8de6c1e846b7da10466873008a5913fb65 Mon Sep 17 00:00:00 2001 From: Alex Domingo Date: Wed, 17 May 2023 15:13:04 +0200 Subject: [PATCH 07/10] mention quota instead of space Co-authored-by: Sam Moors --- source/access/how_to_request_more_quota.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/access/how_to_request_more_quota.rst b/source/access/how_to_request_more_quota.rst index d7ac30a3..56dbad11 100644 --- a/source/access/how_to_request_more_quota.rst +++ b/source/access/how_to_request_more_quota.rst @@ -85,5 +85,5 @@ maximally use. By default, this is set to 50% of the total quota for each user. The sum of all user quotas in your VO can be above 100%. The share for any user indicates what he/she can maximally use, but the actual limit - will then depend on the usage of the other members. The total storage space + will then depend on the usage of the other members. The total storage quota of the VO will always be respected. From 9cad874b89e034d2170b510470bc7e62659a28e2 Mon Sep 17 00:00:00 2001 From: Alex Domingo Date: Mon, 22 May 2023 10:40:51 +0200 Subject: [PATCH 08/10] add site-specific links to request more storage quota --- source/access/how_to_request_more_quota.rst | 41 ++++++++++++--------- 1 file changed, 23 insertions(+), 18 deletions(-) diff --git a/source/access/how_to_request_more_quota.rst b/source/access/how_to_request_more_quota.rst index 56dbad11..8c42f640 100644 --- a/source/access/how_to_request_more_quota.rst +++ b/source/access/how_to_request_more_quota.rst @@ -14,38 +14,43 @@ Before requesting more storage, please check carefully the :ref:`current data usage of your VSC account ` and identify which file system needs a larger quota. +.. _more quota personal: + Personal storage ================ -The following are general guidelines to request more storage in the personal -file systems of your VSC account: +The storage of personal VSC file systems (``VSC_HOME``, ``VSC_DATA`` and +``VSC_SCRATCH``) are managed by your home institute. In case of doubt, please +contact the corresponding :ref:`helpdesk ` for your VSC +account. VSC_HOME The storage of your home directory is small on purpose. If you need more storage, please switch to ``VSC_DATA``. -VSC_DATA - On VSC sites that support :ref:`Virtual Organizations (VO) - `, it is preferable to join a VO to get access to a - larger ``VSC_DATA_VO``. - If that is not the case or if in doubt, please contact :ref:`user support - `. +VSC_DATA, VSC_SCRATCH + Policies on these file systems depend on your home institute: + + * KU Leuven/Hasselt University: follow instructions in + `HPC Service Catalog `_ + * Ghent University: larger storage quotas are preferably provided through + :ref:`Virtual Organizations `. If it is not possible for you + to join a VO, please contact hpc@ugent.be + * University of Antwerp: contact hpc@uantwerpen.be for further information + * Vrije Universiteit Brussel: larger storage quotas are preferably provided through + :ref:`Virtual Organizations `. If it is not possible for you + to join a VO, please contact hpc@vub.be -VSC_SCRATCH - On VSC sites that support :ref:`Virtual Organizations (VO) - `, it is preferable to join a VO to get access to a - larger ``VSC_SCRATCH_VO``. - If that is not the case or if in doubt, please contact :ref:`user support - `. +.. _more quota vo: Virtual Organizations ===================== VSC_DATA_VO, VSC_SCRATCH_VO - The quota of the data storage in your :ref:`Virtual Organization (VO) - ` is managed by the moderator of the VO, who is typically - the leader of your research group. The moderator can manage all quotas of the VO - in the `Edit VO `_ tab of the VSC + The storage quotas of your :ref:`Virtual Organization (VO) ` + are managed by the moderator of the VO, who is typically the leader of your + research group. The moderator can manage all quotas of the VO in the + `Edit VO `_ tab of the VSC account page. Requesting more storage space From b7e0f6e05e0bc1e7051781cceba82a12448a198a Mon Sep 17 00:00:00 2001 From: Alex Domingo Date: Mon, 22 May 2023 10:46:27 +0200 Subject: [PATCH 09/10] add information about requesting more quota for tier-1 project --- source/access/how_to_request_more_quota.rst | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/source/access/how_to_request_more_quota.rst b/source/access/how_to_request_more_quota.rst index 8c42f640..5f56fae0 100644 --- a/source/access/how_to_request_more_quota.rst +++ b/source/access/how_to_request_more_quota.rst @@ -54,7 +54,7 @@ VSC_DATA_VO, VSC_SCRATCH_VO account page. Requesting more storage space -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +----------------------------- VO moderators can request additional quota for ``VSC_DATA_VO`` and ``VSC_SCRATCH_VO``: @@ -72,7 +72,7 @@ VO moderators can request additional quota for ``VSC_DATA_VO`` and ``VSC_SCRATCH #. Your request will be reviewed by the HPC administrators Setting per-member VO quota -~~~~~~~~~~~~~~~~~~~~~~~~~~~ +--------------------------- VO moderators can tweak the share of the VO quota that each member can maximally use. By default, this is set to 50% of the total quota for each user. @@ -92,3 +92,12 @@ maximally use. By default, this is set to 50% of the total quota for each user. for any user indicates what he/she can maximally use, but the actual limit will then depend on the usage of the other members. The total storage quota of the VO will always be respected. + +Tier-1 project +============== + +The scratch storage of your project in tier-1 is managed by the institute +hosting the tier-1 HPC infrastructure. In general, your project will get as +much quota as requested in your application and that should be sufficient to +complete it. If that is not the case, please contact the tier-1 helpdesk at +compute@vscentrum.be. From 02fce840d428520ee48045276f3d217f809ad50e Mon Sep 17 00:00:00 2001 From: Alex Domingo Date: Mon, 22 May 2023 11:14:00 +0200 Subject: [PATCH 10/10] clarify statement on quota assignment to tier-1 project --- source/access/how_to_request_more_quota.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/source/access/how_to_request_more_quota.rst b/source/access/how_to_request_more_quota.rst index 5f56fae0..56d614e0 100644 --- a/source/access/how_to_request_more_quota.rst +++ b/source/access/how_to_request_more_quota.rst @@ -97,7 +97,7 @@ Tier-1 project ============== The scratch storage of your project in tier-1 is managed by the institute -hosting the tier-1 HPC infrastructure. In general, your project will get as -much quota as requested in your application and that should be sufficient to -complete it. If that is not the case, please contact the tier-1 helpdesk at -compute@vscentrum.be. +hosting the tier-1 HPC infrastructure. Your project directory will have the +quota granted to your project, which should be sufficient to complete all +computational tasks in tier-1. If that is not the case, please contact the +tier-1 helpdesk at compute@vscentrum.be.