Skip to content

Commit

Permalink
Import update Mon Nov 20 00:53:15 UTC 2023
Browse files Browse the repository at this point in the history
  • Loading branch information
CallumWalley committed Nov 20, 2023
1 parent 013bc70 commit 01c16e9
Show file tree
Hide file tree
Showing 242 changed files with 4,668 additions and 4,601 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -37,21 +37,21 @@ A quick reminder of our main support channels as well as other sources
of self-service support:

- [Submit a ticket to
Support](https://support.nesi.org.nz/hc/en-gb/requests/new "https://support.nesi.org.nz/hc/en-gb/requests/new") (Note:
non-emergency requests will be addressed on or after 03 January
2024)
Support](https://support.nesi.org.nz/hc/en-gb/requests/new "https://support.nesi.org.nz/hc/en-gb/requests/new") (Note:
non-emergency requests will be addressed on or after 03 January
2024)

- [Sign up for NeSI system status
updates](https://support.nesi.org.nz/hc/en-gb/articles/360000751636 "https://support.nesi.org.nz/hc/en-gb/articles/360000751636") for
advance warning of any system updates or unplanned outages.
updates](https://support.nesi.org.nz/hc/en-gb/articles/360000751636 "https://support.nesi.org.nz/hc/en-gb/articles/360000751636") for
advance warning of any system updates or unplanned outages. 

- [Consult our User
Documentation](https://support.nesi.org.nz/hc/en-gb/categories/360000013836 "https://support.nesi.org.nz/hc/en-gb/categories/360000013836") pages
for instructions and guidelines for using the systems
Documentation](https://support.nesi.org.nz/hc/en-gb/categories/360000013836 "https://support.nesi.org.nz/hc/en-gb/categories/360000013836") pages
for instructions and guidelines for using the systems

- [Visit NeSI’s YouTube
channel](https://www.youtube.com/playlist?list=PLvbRzoDQPkuGMWazx5LPA6y8Ji6tyl0Sp "https://www.youtube.com/playlist?list=PLvbRzoDQPkuGMWazx5LPA6y8Ji6tyl0Sp") for
introductory training webinars
channel](https://www.youtube.com/playlist?list=PLvbRzoDQPkuGMWazx5LPA6y8Ji6tyl0Sp "https://www.youtube.com/playlist?list=PLvbRzoDQPkuGMWazx5LPA6y8Ji6tyl0Sp") for
introductory training webinars

On behalf of the entire NeSI team, we wish you a safe and relaxing
holiday.
holiday. 
Original file line number Diff line number Diff line change
Expand Up @@ -27,12 +27,12 @@ data management policies and best practices for our HPC facilities.
By adopting these measures to regularly audit, clean and manage the
amount of data on our filesystems, we’ll ensure they remain
high-performing and responsive to your research computing workloads and
data science workflows.

data science workflows.

## Upcoming changes to data management processes for project directories

**<u>
**<u>
4-15 October 2021</u>**

The NeSI project filesystem is becoming critically full, however it is
Expand Down Expand Up @@ -63,14 +63,14 @@ and we will consider whether a
[Nearline](https://support.nesi.org.nz/hc/en-gb/articles/360001169956-Long-Term-Storage-Service "https://support.nesi.org.nz/hc/en-gb/articles/360001169956-Long-Term-Storage-Service")
storage allocation would be appropriate to manage this.


 

**18 October 2021**

We will begin a limited roll-out of a new feature to automatically
identify inactive files in  `/nesi/project/` directories and schedule
them for deletion. Generally, we will be looking to identify files that
are inactive / untouched for more than 12 months.
are inactive / untouched for more than 12 months. 

A selection of active projects will be invited to participate in this
first phase of the programme. If you would like to volunteer to be an
Expand All @@ -86,7 +86,7 @@ Alongside this work, we will also adopt a new policy on how long
inactive data may be stored on NeSI systems, particularly once a
research project itself becomes inactive.


 

**<u>January 2022</u>**

Expand All @@ -95,13 +95,13 @@ data management programme to include all active projects on NeSI.
Additional Support documentation and user information sessions will be
hosted prior to wider implementation, to provide advance notice of the
change and to answer any questions you may have around data lifecycle
management.

management. 



## Frequently asked questions

**1) Why are you introducing these new data management processes?
**1) Why are you introducing these new data management processes?
**We want to avoid our online filesystems reaching critically full
levels, as that impacts their performance and availability for users. We
also want to ensure our active storage filesystems aren't being used to
Expand All @@ -110,19 +110,19 @@ for `/nesi/project/` directories will complement our existing programme
of [automatic cleaning of the /nobackup file
system](https://support.nesi.org.nz/hc/en-gb/articles/360001162856 "https://support.nesi.org.nz/hc/en-gb/articles/360001162856").


 

**2) Can I check how much storage I’m currently using on NeSI systems?**

You can query your actual usage and disk allocations at any time using
the following command:
the following command: 

`$ nn_storage_quota`

The values for 'nn\_storage\_quota' are updated approximately every hour
and cached between updates.


 

**3) Can I recover data that I accidentally delete from my /project
directory?**
Expand All @@ -132,7 +132,7 @@ them for up to seven days. For more information, [refer to our File
Recovery
page](https://support.nesi.org.nz/hc/en-gb/articles/360000207315-File-Recovery "https://support.nesi.org.nz/hc/en-gb/articles/360000207315-File-Recovery").


 

**4) Where should I store my data on NeSI systems?**

Expand All @@ -145,9 +145,9 @@ used to build and edit code, provided that the code is under version
control and changes are regularly checked into upstream revision control
systems. The **long-term storage service** should be used for larger
datasets that you only access occasionally and do not need to change in
situ.

situ. 



**5) What should I do if I run out of storage space?**

Expand All @@ -156,7 +156,7 @@ space* and *inodes (number of files)*. If you run into problems with
either of these, [refer to this Support page for more
information](https://support.nesi.org.nz/hc/en-gb/articles/360001125996-I-ve-run-out-of-storage-space "https://support.nesi.org.nz/hc/en-gb/articles/360001125996-I-ve-run-out-of-storage-space").


 

**6) I have questions that aren’t covered here. Who can I talk to?**

Expand All @@ -165,7 +165,7 @@ Support](https://support.nesi.org.nz/hc/en-gb/requests/new "https://support.nesi
No question is too big or small and our intention is always to work with
you to find the best way to manage your research data.


 

## More information

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -55,11 +55,11 @@ We have now recalculated the shares for each pool to take into account
the following:

- The investments into HPC platforms by the various collaborating
institutions and by MBIE;
institutions and by MBIE;
- The capacity of each HPC platform;
- The split of requested time (allocations) by project teams between
the Māui and Mahuika HPC platforms, both overall and within each
institution's pool.
the Māui and Mahuika HPC platforms, both overall and within each
institution's pool.

Under this scheme, any job's priority is affected by the behaviour of
other workload within the same project team, but also other project
Expand All @@ -68,9 +68,9 @@ has been under-using compared to your allocation, your jobs may still be
held up if:

- Other project teams at your institution (within your pool) have been
over-using compared to their allocations, or
over-using compared to their allocations, or
- Your institution has approved project allocations totalling more
time than it is entitled to within its pool's share.
time than it is entitled to within its pool's share.

## What will I notice?

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,37 +31,38 @@ research needs.
**What’s new**

- faster, more powerful computing, enabled by AMD 3rd Gen EPYC Milan
architecture
architecture

- specialised high-memory capabilities, allowing rapid simultaneous
processing
processing

- improved energy efficiency - these nodes are 2.5 times more power
efficient than Mahuika’s original Broadwell nodes
efficient than Mahuika’s original Broadwell nodes

**How to access**

- Visit our Support portal for [instructions to get
started](https://support.nesi.org.nz/hc/en-gb/articles/6367209795471-Milan-Compute-Nodes "https://support.nesi.org.nz/hc/en-gb/articles/6367209795471-Milan-Compute-Nodes")
and details of how the Milan nodes differ from Mahuika’s original
Broadwell nodes
started](https://support.nesi.org.nz/hc/en-gb/articles/6367209795471-Milan-Compute-Nodes "https://support.nesi.org.nz/hc/en-gb/articles/6367209795471-Milan-Compute-Nodes")
and details of how the Milan nodes differ from Mahuika’s original
Broadwell nodes

**Learn more**

- [Watch this webinar](https://youtu.be/IWRZLl__uhg) sharing a quick
overview of the new resources and some tips for making the most of
the nodes.
overview of the new resources and some tips for making the most of
the nodes.

- Bring questions to our [weekly Online Office
Hours](https://support.nesi.org.nz/hc/en-gb/articles/4830713922063-Weekly-Online-Office-Hours "https://support.nesi.org.nz/hc/en-gb/articles/4830713922063-Weekly-Online-Office-Hours")
Hours](https://support.nesi.org.nz/hc/en-gb/articles/4830713922063-Weekly-Online-Office-Hours "https://support.nesi.org.nz/hc/en-gb/articles/4830713922063-Weekly-Online-Office-Hours")

- [Email NeSI
Support](mailto:[email protected] "mailto:[email protected]")
any time

Support](mailto:[email protected] "mailto:[email protected]")
any time



If you have feedback on the new nodes or suggestions for improving your
experience getting started with or using any of our systems, please [get
in touch](mailto:[email protected] "mailto:[email protected]").


Original file line number Diff line number Diff line change
Expand Up @@ -24,10 +24,10 @@ zendesk_section_id: 200732737
[//]: <> (^^^^^^^^^^^^^^^^^^^^)
[//]: <> (REMOVE ME IF PAGE VALIDATED)

A Slurm configuration change has been made on Mahuika so that the
A Slurm configuration change has been made on Mahuika so that the 
maximum size of [core
file](https://support.nesi.org.nz/hc/en-gb/articles/360001584875-What-is-a-core-file-) that
can be generated inside a job now defaults to `0` bytes rather
than `unlimited`.
than `unlimited`. 

You can reenable core dumps with `ulimit -c unlimited` .
38 changes: 19 additions & 19 deletions docs/General/Announcements/Maui_upgrade_is_complete.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,33 +53,33 @@ rebuilt and/or updated versions of these applications (though this will
be an ongoing effort post-upgrade).

The following information will help your transition from the pre-upgrade
Māui environment to the post-upgrade one:
Māui environment to the post-upgrade one: 

- The three main toolchains (CrayCCE, CrayGNU and CrayIntel) have all
been updated to release 23.02 (CrayCCE and CrayGNU) and 23.02-19
(CrayIntel). **The previously installed versions are no longer
available**.
been updated to release 23.02 (CrayCCE and CrayGNU) and 23.02-19
(CrayIntel). **The previously installed versions are no longer
available**.
- Consequently, nearly all of the previously provided **environment
modules have been replaced by new versions**. You can use the
*module avail* command to see what versions of those software
packages are now available. If your batch scripts load exact module
versions, they will need updating.
modules have been replaced by new versions**. You can use the
*module avail* command to see what versions of those software
packages are now available. If your batch scripts load exact module
versions, they will need updating.
- The few jobs in the Slurm queue at the start of the upgrade process
have been placed in a “user hold” state. You have the choice of
cancelling them with *scancel &lt;jobid&gt;* or releasing them with
*scontrol release &lt;jobid&gt;*.
have been placed in a “user hold” state. You have the choice of
cancelling them with *scancel &lt;jobid&gt;* or releasing them with
*scontrol release &lt;jobid&gt;*.
- Be aware that if you have jobs submitted that rely on any software
built before the upgrade, there is a good chance that this software
will not run. **We recommend rebuilding any binaries you maintain**
before running jobs that utilise those binaries.
built before the upgrade, there is a good chance that this software
will not run. **We recommend rebuilding any binaries you maintain**
before running jobs that utilise those binaries.
- Note that Māui login does not require adding a second factor to the
password when authenticating on the Māui login node after the first
successful login attempt. That is, if you have successfully logged
in using &lt;first factor&gt;&lt;second factor&gt; format, no second
factor part will be required later on.
password when authenticating on the Māui login node after the first
successful login attempt. That is, if you have successfully logged
in using &lt;first factor&gt;&lt;second factor&gt; format, no second
factor part will be required later on.

We have also updated our support documentation for Māui to reflect the
changes, so please review it before starting any new projects.
changes, so please review it before starting any new projects. 

## Software Changes

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ zendesk_section_id: 200732737
We’re excited to announce an addition of new GPU capabilities to our
platform and some noteworthy changes to resource pricing as a result.


 

**New Graphics Processing Units (GPUs)**

Expand All @@ -32,7 +32,7 @@ providing a significant boost in computing performance and an
environment particularly suited to machine learning workloads. Over the
last few months we’ve worked directly with a group of beta tester
researchers to ensure this new capability is fit-for-purpose and tuned
to communities' specific software and tool requirements.
to communities' specific software and tool requirements. 

These new A100s, alongside [software optimised for data
science](https://support.nesi.org.nz/hc/en-gb/articles/360004558895-What-software-environments-on-NeSI-are-optimised-for-Machine-Learning-approaches-),
Expand All @@ -41,7 +41,7 @@ this is you, [contact NeSI
Support](mailto:https://support.nesi.org.nz/hc/en-gb/requests/new) to
discuss how these new resources could support your work.


 

**Reduced pricing for P100s**

Expand All @@ -65,7 +65,7 @@ you have questions about allocations or how to access the P100s,
[contact NeSI
Support](mailto:https://support.nesi.org.nz/hc/en-gb/requests/new).


 

**Sharing our learning along the way**

Expand All @@ -81,7 +81,7 @@ conducted in the spaces of deep learning and molecular dynamics codes,
as well as take a closer look at which codes are suitable to run on GPUs
and whether your research project is a fit.


 

**Future GPU investments**

Expand All @@ -99,13 +99,13 @@ A100s for something other than machine learning, let us know by
Support](mailto:https://support.nesi.org.nz/hc/en-gb/requests/new) -
that way we can keep you up to date on our plans.


 

If you have questions or comments on anything mentioned above,
please [get in
touch](https://support.nesi.org.nz/hc/en-gb/requests/new).


 

Thank you,

Expand Down
Loading

0 comments on commit 01c16e9

Please sign in to comment.