Not initiating instances in hpc-slurm (if nodes > 6)

[cpcpython]

Hi List

I was using hpc-slurm (default yaml) and it was OK until I use compute nodes up to 6.
Pls note that when I use nodes=6, I have 36/120 cores are used. So it is not the problem of Quotas.

But When I use increase compute nodes, say 8, a node was not created and hence the creation of HPC system were failed.
Attached below is the command to create compute nodes and its results.

Is any solutions for this error?
Thanks in advance

Km

$ srun -N 2 -p compute hostname
hpcslurm-computenodeset-0
hpcslurm-computenodeset-1

$ srun -N 4 -p compute hostname
hpcslurm-computenodeset-1
hpcslurm-computenodeset-2
hpcslurm-computenodeset-3
hpcslurm-computenodeset-0


$ srun -N 6 -p compute hostname
hpcslurm-computenodeset-5
hpcslurm-computenodeset-0
hpcslurm-computenodeset-4
hpcslurm-computenodeset-3
hpcslurm-computenodeset-1
hpcslurm-computenodeset-2

$ srun -N 8 -p compute hostname
srun: error: Node failure on hpcslurm-computenodeset-6
srun: error: Nodes hpcslurm-computenodeset-[0-7] are still not ready
srun: error: Something is wrong with the boot of the nodes.

$ srun -N 8 -p compute hostname
srun: error: Node failure on hpcslurm-computenodeset-4
srun: error: Nodes hpcslurm-computenodeset-[0-7] are still not ready
srun: error: Something is wrong with the boot of the nodes.

$ sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
compute      up   infinite      7  idle% hpcslurm-computenodeset-[0-3,5-7]
compute      up   infinite     12  idle~ hpcslurm-computenodeset-[8-19]
compute      up   infinite      1  down~ hpcslurm-computenodeset-4
debug*       up   infinite      4  idle~ hpcslurm-debugnodeset-[0-3]
h3           up   infinite     20  idle~ hpcslurm-h3nodeset-[0-19]

[mr0re1]

Hi @cpcpython , could you please inspect /slurm/scripts/log/resume.log , does it contain any errors?
You can also take a look at the instance hpcslurm-computenodeset-6 (using CloudLogging), was it created, if yes, where there any errors in startup script?

[cpcpython]

Dear Ivan,
Thanks for the mail, I could fixed it by increasing

1. C2_CPU quotas to 180
2. Network quotas to 30 from 15.

Now its fine!

best regards

Km