Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Abaqus wip #30

Merged
merged 3 commits into from
Nov 26, 2023
Merged

Abaqus wip #30

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f63b8c8
Testing editing on my phone
CallumWalley Nov 21, 2023
8e6e2e6
Update ABAQUS.md
CallumWalley Nov 21, 2023
0aa483f
Done I think
CallumWalley Nov 23, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
271 changes: 113 additions & 158 deletions docs/Scientific_Computing/Supported_Applications/ABAQUS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---

Check notice on line 1 in docs/Scientific_Computing/Supported_Applications/ABAQUS.md

View workflow job for this annotation

GitHub Actions / runChecks / metacheck

title.mismatch 

Title in filename('Abaqus') doesn't match that set in meta ('ABAQUS')

Check notice on line 1 in docs/Scientific_Computing/Supported_Applications/ABAQUS.md

View workflow job for this annotation

GitHub Actions / runChecks / metacheck

title.mismatch 

Title in meta('ABAQUS') doesn't match that set in filename ('Abaqus')
created_at: '2015-10-12T00:28:38Z'
hidden: false
position: 21
Expand All @@ -8,197 +8,159 @@
- gpu
- mpi
- omp
- fea
title: ABAQUS
vote_count: 2
vote_sum: 0
zendesk_article_id: 212457807
zendesk_section_id: 360000040076
---

A list of ABAQUS commands can be found with:


[//]: <> (REMOVE ME IF PAGE VALIDATED)
[//]: <> (vvvvvvvvvvvvvvvvvvvv)
!!! warning
This page has been automatically migrated and may contain formatting errors.
[//]: <> (^^^^^^^^^^^^^^^^^^^^)
[//]: <> (REMOVE ME IF PAGE VALIDATED)

<!-- The above lines, specifying the category, section and title, must be
present and always comprising the first three lines of the article. -->

A list of commands can be found with:

``` sl
``` sh
abaqus help
```

## Licences

[Hyperthreading](https://support.nesi.org.nz/hc/en-gb/articles/360000568236)
can provide significant speedup to your computations, however
hyperthreaded CPUs will use twice the number of licence tokens. It may
be worth adding  `#SBATCH --hint nomultithread` to your slurm script if
licence tokens are your main limiting factor.
!!! prerequisite Tips

!!! note
Required ABAQUS licences can be determined by this simple and

Check warning on line 34 in docs/Scientific_Computing/Supported_Applications/ABAQUS.md

View workflow job for this annotation

GitHub Actions / runChecks / proselint

leonard.exclamation.multiple 

'Stop yelling. Keep your exclamation points under control. Found once elsewhere.'
intuitive formula `⌊ 5 x N`<sup>`0.422`</sup>` ⌋` where `N` is number
of CPUs.

You can force ABAQUS to use a specific licence type by setting the
parameter `academic=TEACHING` or `academic=RESEARCH` in a relevant
[environment file](#env_file).

Check warning on line 40 in docs/Scientific_Computing/Supported_Applications/ABAQUS.md

View workflow job for this annotation

GitHub Actions / runChecks / mdlint

Link fragments should be valid 

None

## Solver Compatibility

Not all solvers are compatible with all types of parallelisation.

| | | | | |
|-------------------|--------------------|------------------|---------------|----------------|
|   | Element operations | Iterative solver | Direct solver | Lanczos solver |
| `mp_mode=threads` | ✖ | ✔ | ✔ | ✔ |
| `mp_mode=mpi` | ✔ | ✔ | ✖ | ✖ |
!!! prerequisite Note
| | | | | |
|-------------------|--------------------|-----------|--------|---------|
|   | Element operations | Iterative | Direct | Lanczos |
| `mp_mode=threads` | ✖ | ✔ | ✔ | ✔ |
| `mp_mode=mpi` | ✔ | ✔ | ✖ | ✖ |

!!! warning
If your input files were created using an older version of ABAQUS you
will need to update them using the command,
``` sl
``` sh
abaqus -upgrade -job new_job_name -odb old.odb
```
or
``` sl
``` sh
abaqus -upgrade -job new_job_name -inp old.inp
```

<table>
<colgroup>
<col style="width: 50%" />
<col style="width: 50%" />
</colgroup>
<tbody>
<tr class="odd">
<td style="width: 506px"><h2 id="serial">Serial</h2>
<hr />
<p>For when only <span>one CPU is required</span>, generally as part of
an <a
href="https://support.nesi.org.nz/hc/en-gb/articles/360000690275-Parallel-Execution#t_array">job
array</a>.</p>
<p> </p></td>
<td style="width: 163px"><div class="sourceCode" id="cb1"><pre
class="sourceCode bash"><code class="sourceCode bash"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="co">#!/bin/bash -e</span></span>
<span id="cb1-2"><a href="#cb1-2" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb1-3"><a href="#cb1-3" aria-hidden="true" tabindex="-1"></a><span class="co">#SBATCH --job-name ABAQUS-Shared</span></span>
<span id="cb1-4"><a href="#cb1-4" aria-hidden="true" tabindex="-1"></a><span class="co">#SBATCH --time 00:05:00 # Walltime</span></span>
<span id="cb1-5"><a href="#cb1-5" aria-hidden="true" tabindex="-1"></a><span class="co">#SBATCH --cpus-per-task 1 </span></span>
<span id="cb1-6"><a href="#cb1-6" aria-hidden="true" tabindex="-1"></a><span class="co">#SBATCH --mem 1500 # total mem</span></span>
<span id="cb1-7"><a href="#cb1-7" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb1-8"><a href="#cb1-8" aria-hidden="true" tabindex="-1"></a><span class="ex">module</span> load ABAQUS/2019</span>
<span id="cb1-9"><a href="#cb1-9" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb1-10"><a href="#cb1-10" aria-hidden="true" tabindex="-1"></a><span class="ex">abaqus</span> job=<span class="st">&quot;propeller_s4rs_c3d8r&quot;</span> verbose=2 interactive </span></code></pre></div></td>
</tr>
<tr class="even">
<td style="width: 506px"><h2 id="shared-memory">Shared Memory</h2>
<hr />
<code class="sl">mp_mode=threads</code>
<p>Uses a nodes shared memory for communication. </p>
<p>May have a small speedup compared to MPI when using a low number of
CPUs, scales poorly. Needs significantly less memory than MPI.</p>
<em>Hyperthreading may be enabled if using shared memory but it is not
recommended.</em></td>
<td style="width: 163px"><div class="sourceCode" id="cb2"><pre
class="sourceCode bash"><code class="sourceCode bash"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a><span class="co">#!/bin/bash -e</span></span>
<span id="cb2-2"><a href="#cb2-2" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb2-3"><a href="#cb2-3" aria-hidden="true" tabindex="-1"></a><span class="co">#SBATCH --job-name ABAQUS-Shared</span></span>
<span id="cb2-4"><a href="#cb2-4" aria-hidden="true" tabindex="-1"></a><span class="co">#SBATCH --time 00:05:00 # Walltime</span></span>
<span id="cb2-5"><a href="#cb2-5" aria-hidden="true" tabindex="-1"></a><span class="co">#SBATCH --cpus-per-task 4 </span></span>
<span id="cb2-6"><a href="#cb2-6" aria-hidden="true" tabindex="-1"></a><span class="co">#SBATCH --mem 2G # total mem</span></span>
<span id="cb2-7"><a href="#cb2-7" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb2-8"><a href="#cb2-8" aria-hidden="true" tabindex="-1"></a><span class="ex">module</span> load ABAQUS/2019</span>
<span id="cb2-9"><a href="#cb2-9" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb2-10"><a href="#cb2-10" aria-hidden="true" tabindex="-1"></a><span class="ex">abaqus</span> job=<span class="st">&quot;propeller_s4rs_c3d8r&quot;</span> verbose=2 interactive <span class="dt">\</span></span>
<span id="cb2-11"><a href="#cb2-11" aria-hidden="true" tabindex="-1"></a> cpus=<span class="va">${SLURM_CPUS_PER_TASK}</span> mp_mode=threads</span></code></pre></div></td>
</tr>
<tr class="odd">
<td style="width: 506px"><h2 id="udf">UDF</h2>
<hr />
<p>Shared memory run with user defined function (fortran or C). </p>
<p><code class="sl">user=&lt;name_of_function&gt;</code> </p>
<p>Function will be compiled at start of run. </p>
<p><em>You may need to chance the function suffix if you usually compile
on windows.</em></p></td>
<td style="width: 163px"><div class="sourceCode" id="cb3"><pre
class="sourceCode bash"><code class="sourceCode bash"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true" tabindex="-1"></a><span class="co">#!/bin/bash -e</span></span>
<span id="cb3-2"><a href="#cb3-2" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb3-3"><a href="#cb3-3" aria-hidden="true" tabindex="-1"></a><span class="co">#SBATCH --job-name ABAQUS-SharedUDF</span></span>
<span id="cb3-4"><a href="#cb3-4" aria-hidden="true" tabindex="-1"></a><span class="co">#SBATCH --time 00:05:00 # Walltime</span></span>
<span id="cb3-5"><a href="#cb3-5" aria-hidden="true" tabindex="-1"></a><span class="co">#SBATCH --cpus-per-task 4 </span></span>
<span id="cb3-6"><a href="#cb3-6" aria-hidden="true" tabindex="-1"></a><span class="co">#SBATCH --mem 2G # total mem</span></span>
<span id="cb3-7"><a href="#cb3-7" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb3-8"><a href="#cb3-8" aria-hidden="true" tabindex="-1"></a><span class="ex">module</span> load imkl</span>
<span id="cb3-9"><a href="#cb3-9" aria-hidden="true" tabindex="-1"></a><span class="ex">module</span> load ABAQUS/2019</span>
<span id="cb3-10"><a href="#cb3-10" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb3-11"><a href="#cb3-11" aria-hidden="true" tabindex="-1"></a><span class="ex">abaqus</span> job=<span class="st">&quot;propeller_s4rs_c3d8r&quot;</span> user=my_udf.f90 verbose=2 interactive <span class="dt">\</span></span>
<span id="cb3-12"><a href="#cb3-12" aria-hidden="true" tabindex="-1"></a> cpus=<span class="va">${SLURM_CPUS_PER_TASK}</span> mp_mode=threads</span></code></pre></div></td>
</tr>
<tr class="even">
<td class="wysiwyg-text-align-left" style="width: 506px"><h2
id="distributed-memory">Distributed Memory</h2>
<hr />
<code class="sl">mp_mode=mpi</code>
<p>Multiple <em>processes</em> each with a single <em>thread</em>.</p>
<p>Not limited to <span>one node</span>.<br />
Model will be segmented into <code class="sl">-np</code> pieces which
should be equal to <code class="sl">--ntasks</code>.</p>
<p>Each task could be running on a different node leading to increased
communication overhead<br />
.Jobs can be limited to a single node by adding  <code
class="sl">--nodes=1</code> however this will increase your time in the
queue as contiguous cpu's are harder to schedule.</p>
<p>This is the default method if <code class="sl">mp_mode</code> is left
unspecified.</p></td>
<td style="width: 163px"><div class="sourceCode" id="cb4"><pre
class="sourceCode bash"><code class="sourceCode bash"><span id="cb4-1"><a href="#cb4-1" aria-hidden="true" tabindex="-1"></a><span class="co">#!/bin/bash -e</span></span>
<span id="cb4-2"><a href="#cb4-2" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb4-3"><a href="#cb4-3" aria-hidden="true" tabindex="-1"></a><span class="co">#SBATCH --job-name ABAQUS-Distributed </span></span>
<span id="cb4-4"><a href="#cb4-4" aria-hidden="true" tabindex="-1"></a><span class="co">#SBATCH --time 00:05:00 # Walltime</span></span>
<span id="cb4-5"><a href="#cb4-5" aria-hidden="true" tabindex="-1"></a><span class="co">#SBATCH --ntasks 8 </span></span>
<span id="cb4-6"><a href="#cb4-6" aria-hidden="true" tabindex="-1"></a><span class="co">#SBATCH --mem-per-cpu 1500 # Each CPU needs it&#39;s own.</span></span>
<span id="cb4-7"><a href="#cb4-7" aria-hidden="true" tabindex="-1"></a><span class="co">#SBATCH --nodes 1</span></span>
<span id="cb4-8"><a href="#cb4-8" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb4-9"><a href="#cb4-9" aria-hidden="true" tabindex="-1"></a><span class="ex">module</span> load ABAQUS/2019</span>
<span id="cb4-10"><a href="#cb4-10" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb4-11"><a href="#cb4-11" aria-hidden="true" tabindex="-1"></a><span class="ex">abaqus</span> job=<span class="st">&quot;propeller_s4rs_c3d8r&quot;</span> verbose=2 interactive <span class="dt">\</span></span>
<span id="cb4-12"><a href="#cb4-12" aria-hidden="true" tabindex="-1"></a> cpus=<span class="va">${SLURM_NTASKS}</span> mp_mode=mpi</span></code></pre></div></td>
</tr>
<tr class="odd">
<td style="width: 506px"><h2 id="gpus">GPUs</h2>
<hr />
<p>The GPU nodes are limited to <span>16 CPUs</span></p>
<p>In order for the GPUs to be worthwhile, you should see a speedup
equivalent to <span>56 CPU</span>'s per GPU used. GPU modes will
generally have less memory/cpus</p></td>
<td style="width: 163px"><div class="sourceCode" id="cb5"><pre
class="sourceCode bash"><code class="sourceCode bash"><span id="cb5-1"><a href="#cb5-1" aria-hidden="true" tabindex="-1"></a><span class="co">#!/bin/bash -e</span></span>
<span id="cb5-2"><a href="#cb5-2" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb5-3"><a href="#cb5-3" aria-hidden="true" tabindex="-1"></a><span class="co">#SBATCH --job-name ABAQUS-gpu</span></span>
<span id="cb5-4"><a href="#cb5-4" aria-hidden="true" tabindex="-1"></a><span class="co">#SBATCH --time 00:05:00 # Walltime</span></span>
<span id="cb5-5"><a href="#cb5-5" aria-hidden="true" tabindex="-1"></a><span class="co">#SBATCH --cpus-per-task 4 </span></span>
<span id="cb5-6"><a href="#cb5-6" aria-hidden="true" tabindex="-1"></a><span class="co">#SBATCH --mem 4G # total mem</span></span>
<span id="cb5-7"><a href="#cb5-7" aria-hidden="true" tabindex="-1"></a><span class="co">#SBATCH --gpus-per-node 1</span></span>
<span id="cb5-8"><a href="#cb5-8" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb5-9"><a href="#cb5-9" aria-hidden="true" tabindex="-1"></a><span class="ex">module</span> load ABAQUS/2019 </span>
<span id="cb5-10"><a href="#cb5-10" aria-hidden="true" tabindex="-1"></a><span class="ex">module</span> load CUDA</span>
<span id="cb5-11"><a href="#cb5-11" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb5-12"><a href="#cb5-12" aria-hidden="true" tabindex="-1"></a><span class="ex">abaqus</span> job=<span class="st">&quot;propeller_s4rs_c3d8r&quot;</span> verbose=2 interactive <span class="dt">\</span></span>
<span id="cb5-13"><a href="#cb5-13" aria-hidden="true" tabindex="-1"></a> cpus=<span class="va">${SLURM_CPUS_PER_TASK}</span> gpus=<span class="va">${SLURM_GPUS_PER_NODE}</span> mp_mode=threads</span></code></pre></div></td>
</tr>
</tbody>
</table>

Check warning on line 62 in docs/Scientific_Computing/Supported_Applications/ABAQUS.md

View workflow job for this annotation

GitHub Actions / runChecks / mdlint

Trailing spaces 

Expected: 0 or 2; Actual: 5
## Examples

=== Serial
For when only one CPU is required, generally as part of
a [job array](https://support.nesi.org.nz/hc/en-gb/articles/360000690275-Parallel-Execution#t_array)
```
#!/bin/bash -e
#SBATCH --job-name ABAQUS-serial
#SBATCH --time 00:05:00 # Walltime
#SBATCH --cpus-per-task 1

Check warning on line 72 in docs/Scientific_Computing/Supported_Applications/ABAQUS.md

View workflow job for this annotation

GitHub Actions / runChecks / mdlint

Trailing spaces 

Expected: 0 or 2; Actual: 1
#SBATCH --mem 1500 # total mem
module load ABAQUS/{{applications.ABAQUS.machines.mahuika.versions | last}}
abaqus job="propeller_s4rs_c3d8r" verbose=2 interactive
```
=== "Shared Memory"
`mp_mode=threads`
Uses a nodes shared memory for communication.
May have a small speedup compared to MPI when using a low number of
CPUs, scales poorly. Needs significantly less memory than MPI.
Hyperthreading may be enabled if using shared memory but it is not
recommended.

Check warning on line 84 in docs/Scientific_Computing/Supported_Applications/ABAQUS.md

View workflow job for this annotation

GitHub Actions / runChecks / mdlint

Trailing spaces 

Expected: 0 or 2; Actual: 4
```sl

Check warning on line 85 in docs/Scientific_Computing/Supported_Applications/ABAQUS.md

View workflow job for this annotation

GitHub Actions / runChecks / mdlint

Code block style 

Expected: fenced; Actual: indented
#!/bin/bash -e
#SBATCH --job-name ABAQUS-Shared
#SBATCH --time 00:05:00 # Walltime
#SBATCH --cpus-per-task 4
#SBATCH --mem 2G # total mem
module load ABAQUS/{{applications.ABAQUS.machines.mahuika.versions}}
abaqus job="propeller_s4rs_c3d8r verbose=2 interactive \
cpus=${SLURM_CPUS_PER_TASK} mp_mode=threads

=== "UDF"

Check warning on line 95 in docs/Scientific_Computing/Supported_Applications/ABAQUS.md

View workflow job for this annotation

GitHub Actions / runChecks / proselint

typography.symbols.curly_quotes 

'Use curly quotes “”, not straight quotes "". Found 8 times elsewhere.'
Shared memory run with user defined function (fortran or C).
Function will be compiled at start of run.
You may need to chance the function suffix if you usually compile
on windows.


Check warning on line 101 in docs/Scientific_Computing/Supported_Applications/ABAQUS.md

View workflow job for this annotation

GitHub Actions / runChecks / mdlint

Multiple consecutive blank lines 

Expected: 1; Actual: 2
```

Check warning on line 102 in docs/Scientific_Computing/Supported_Applications/ABAQUS.md

View workflow job for this annotation

GitHub Actions / runChecks / mdlint

Code block style 

Expected: fenced; Actual: indented
#!/bin/bash -e

#SBATCH --job-name ABAQUS-SharedUDF
#SBATCH --time 00:05:00 # Walltime
#SBATCH --cpus-per-task 4
#SBATCH --mem 2G # total mem

module load imkl</span>
module</span> load ABAQUS/{{{{applications.ABAQUS.machines.mahuika.versions | last}}
abaqus job="propeller_s4rs_c3d8r" user=my_udf.f90 \
verbose=2 interactive cpus=${SLURM_CPUS_PER_TASK} mp_mode=threads
=== "Distributed Memory"
`mp_mode=mpi`
Multiple processes each with a single *thread*. Not limited to one node.
Model will be segmented into `-np` pieces which
should be equal to `--ntasks`
Each task could be running on a different node leading to increased
communication overhead. Jobs can be limited to a single node by adding `--nodes=1` however this will increase your time in the
queue as contiguous cpu's are harder to schedule.
This is the default method if `mp_mode` is left
unspecified.

Check warning on line 124 in docs/Scientific_Computing/Supported_Applications/ABAQUS.md

View workflow job for this annotation

GitHub Actions / runChecks / mdlint

Trailing spaces 

Expected: 0 or 2; Actual: 4
```sl

Check warning on line 125 in docs/Scientific_Computing/Supported_Applications/ABAQUS.md

View workflow job for this annotation

GitHub Actions / runChecks / mdlint

Code block style 

Expected: fenced; Actual: indented
#!/bin/bash -e
#SBATCH --job-name ABAQUS-Distributed
#SBATCH --time 00:05:00 # Walltime</span></span>
#SBATCH --ntasks 8
#SBATCH --mem-per-cpu 1500 # Each CPU needs it&#39;s own.
#SBATCH --nodes 1

module load ABAQUS/{{applications.ABAQUS.machines.mahuika.versions | last}}
abaqus job "propeller_s4rs_c3d8r" verbose=2 interactive cpus=${SLURM_NTASKS} mp_mode=mpi
```

=== "GPUs"
The GPU nodes are limited to 16 CPUs
In order for the GPUs to be worthwhile, you should see a speedup
equivalent to 56 CPU's per GPU used. GPU modes will
generally have less memory/cpus.

```

Check warning on line 143 in docs/Scientific_Computing/Supported_Applications/ABAQUS.md

View workflow job for this annotation

GitHub Actions / runChecks / mdlint

Code block style 

Expected: fenced; Actual: indented
#!/bin/bash -e

#SBATCH --job-name ABAQUS-gpu
#SBATCH --time 00:05:00 # Walltime</span></span>
#SBATCH --cpus-per-task 4
#SBATCH --mem 4G # total mem</span></span>
#SBATCH --gpus-per-node
module load ABAQUS/{{applications.ABAQUS.machines.mahuika.versions | last}}
module load CUDA
abaqus job="propeller_s4rs_c3d8r" verbose=2 interactive \
cpus=${SLURM_CPUS_PER_TASK} gpus=${SLURM_GPUS_PER_NODE} mp_mode=threads
```

## User Defined Functions 

User defined functions (UDFs) can be included on the command line with
the argument `user=<filename>` where `<filename>` is the C or fortran
source code.

Extra compiler options can be set in your local `abaqus_v6.env` file.
Extra compiler options can be set in your local `abaqus_v6.env` [file](#environment_file).

The default compile commands are for `imkl`, other compilers can be
loaded with `module load`, you may have to change the [compile
Expand All @@ -212,7 +174,7 @@
a number of parameters that define how the your job will run, some of
these you may with to change.

These parameters are read, 
These parameters are read in the following order of preference, 

`../ABAQUS/SMA/site/abaqus_v6.env` Set by NeSI and cannot be changed.

Expand All @@ -234,15 +196,8 @@
# After job is finished.
rm "abaqus_v6.env"
```
!!! prerequisite Useful Links
- [Command line options for standard
submission.](https://www.sharcnet.ca/Software/Abaqus610/Documentation/docs/v6.10/books/usb/default.htm?startat=pt01ch03s02abx02.html)



![ABAQUS\_speedup\_SharedVMPI.png](../../assets/images/ABAQUS.png)



*Note: Hyperthreading off, testing done on small mechanical FEA model.
Results highly model dependant. Do your own tests.*
Results highly model dependant. Do your own tests.*
Loading