Maximum resources possible for allocation

[mdamgaard-biolib]

I have searched through the documentation and made some experiments.
Yet i have not figured out what the maximum amount of vCPUs and memory i could allocate to an enclave is.

Have you done these test at some point and reached a value or is it more like above some point behaviour becomes "undefined"?

For example:
An r5.24xlarge ec2 instance boasts 96vCPUs and 728GB RAM, so i was wondering how many of those resources i could realistically use in an enclave.

[exFalso]

I'd recommend looking at the nitro-cli-config logic for allocating enclave CPUs, in particular the following function and its comment:

aws-nitro-enclaves-cli/bootstrap/nitro-cli-config

Lines 544 to 549 in 8f6ed74

	# Configure the CPU pool using the provided CPU count.
	# Auto-generate a CPU pool given the following conditions:
	# * All the CPUs need to be from the same NUMA node.
	# * CPU 0 and its siblings need to remain available to the primary / parent VM.
	# * Full CPU core(s) need(s) to be included in the CPU pool.
	function configure_cpu_pool_by_cpu_count {

So, vcpu 0 must be retained for host usage + one whole NUMA must be allocated(and at most one) for the enclave. For smaller VMs I have tried there have been 2 NUMAs, and the vCPUs are evenly distributed between them. This means that at most half of the overall vCPUs can be allocated to enclave usage.

To check the vCPU - NUMA assignment you can use lscpu -p=cpu,node. Note that this will only display online CPUs. Once CPUs are allocated to an enclave they are brought offline, so they won't be displayed.

[lx3-g]

For r5.24xlarge ec2 instance that has 728GB RAM, when in /etc/nitro_enclaves/allocator.yaml
I set memory_mib to 367572 (a little less than half the RAM) and then run
systemctl start nitro-enclaves-allocator.service
It works fine.
When I set it to 397572 (a little over the half), it fails with the error below.
What is the percentage of the overall memory that the enclave machine can use? Is there a hard limit, or is it always percentage?

● nitro-enclaves-allocator.service - Nitro Enclaves Resource Allocator
   Loaded: loaded (/usr/lib/systemd/system/nitro-enclaves-allocator.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Mon 2023-10-30 17:00:43 UTC; 11s ago
  Process: 19705 ExecStart=/usr/bin/nitro-enclaves-allocator (code=exited, status=1/FAILURE)
 Main PID: 19705 (code=exited, status=1/FAILURE)

Oct 30 17:00:30 ip-10-0-235-54.ec2.internal nitro-enclaves-allocator[19705]: Will try to reserve 667572 MB of memory on node 1.
Oct 30 17:00:30 ip-10-0-235-54.ec2.internal nitro-enclaves-allocator[19705]: Configuring the huge page memory...
Oct 30 17:00:34 ip-10-0-235-54.ec2.internal nitro-enclaves-allocator[19705]: - Reserved 368 pages of type: 1048576kB.
Oct 30 17:00:34 ip-10-0-235-54.ec2.internal nitro-enclaves-allocator[19705]: - Reserved 2677 pages of type: 2048kB.
Oct 30 17:00:34 ip-10-0-235-54.ec2.internal nitro-enclaves-allocator[19705]: Memory configuration failed, rolling back memory reservations...
Oct 30 17:00:43 ip-10-0-235-54.ec2.internal nitro-enclaves-allocator[19705]: Error: Failed to configure entire amount of requested memory. This indicates insufficient system resources.
Oct 30 17:00:43 ip-10-0-235-54.ec2.internal systemd[1]: nitro-enclaves-allocator.service: main process exited, code=exited, status=1/FAILURE
Oct 30 17:00:43 ip-10-0-235-54.ec2.internal systemd[1]: Failed to start Nitro Enclaves Resource Allocator.
Oct 30 17:00:43 ip-10-0-235-54.ec2.internal systemd[1]: Unit nitro-enclaves-allocator.service entered failed state.
Oct 30 17:00:43 ip-10-0-235-54.ec2.internal systemd[1]: nitro-enclaves-allocator.service failed.