fix nccl test eks

[roywei]

Issue #, if available:

Description of changes:

1. Fixed image pull policy config, to resolve this error during deployment

Error from server (BadRequest): error when creating "nccl-tests.yaml": MPIJob in version "v2beta1" cannot be handled as a MPIJob: strict decoding error: unknown field "spec.mpiReplicaSpecs.Launcher.template.spec.imagePullPolicy", unknown field "spec.mpiReplicaSpecs.Worker.template.spec.imagePullPolicy"  yaml file apiVersion: 

2. removed unnecessary flags for EFA, these are auto-set in OFI plugin now.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[mhuguesaws]

Keep those variable to be consistent with slurm.

[mhuguesaws]

@roywei do you plan to update this PR?

[roywei]

Hi @mhuguesaws thanks for the review and sorry for late reply. I think these flags are not needed anymore per OFI docs, no matter if this is on slurm or k8s. Also if you set FI_EFA_USE_DEVICE_RDMA on some old p3 instances with new OFI and libfabric version if may cause crash. See here https://github.com/aws/aws-ofi-nccl/blob/master/doc/efa-env-var.md

[mhuguesaws]

This test is specific to p5. Please keep as it is or make it consistent in Slurm as well. Thank you

[mhuguesaws]

Left comments

[roywei]

let me know if i missed anything about the flags. I can also update the slurm nccl-tests flags if needed

[roywei]

Hi @mhuguesaws thanks for the review and sorry for late reply. I think these flags are not needed anymore per OFI docs, no matter if this is on slurm or k8s. Also if you set FI_EFA_USE_DEVICE_RDMA on some old p3 instances with new OFI and libfabric version if may cause crash. See here https://github.com/aws/aws-ofi-nccl/blob/master/doc/efa-env-var.md