Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a GitHub Action to run Packer builds for Windows AMI #5992

Open
zxiiro opened this issue Nov 27, 2024 · 5 comments
Open

Create a GitHub Action to run Packer builds for Windows AMI #5992

zxiiro opened this issue Nov 27, 2024 · 5 comments
Assignees

Comments

@zxiiro
Copy link
Collaborator

zxiiro commented Nov 27, 2024

Currently the Windows AMI is being manually created and uploaded to both Meta and PyTorch Foundation AWS accounts. Instead we should utilize GitHub Actions to create a publish job that can handle this automatically for us.

@zxiiro zxiiro self-assigned this Nov 27, 2024
@zxiiro
Copy link
Collaborator Author

zxiiro commented Dec 3, 2024

This task will require more infrastructure setup than I initially thought. So the current ALI system deployed in PyTorch Foundation's AWS Account is a repository level ALI deployment so it only works with pytorch/pytorch meaning we currently don't have a way to launch instances from the LF fleet against any other repository.

@malfet
Copy link
Contributor

malfet commented Dec 9, 2024

Looks like we need to create some sort of policies thru AWS accounts to allow both account push AMIs
@zxiiro , @ZainRizvi , @atalman do you mind leaving the comment what is blocking you at the moment?

@zxiiro
Copy link
Collaborator Author

zxiiro commented Dec 9, 2024

Originally I was thinking that we'd run 2 jobs one on Meta and one on LF side to push the AMI automatically. But on the CI Sync meeting today there were a few other suggestions.

  1. Allow Meta runners to deploy the Windows AMI in LF Account
  2. Meta account shares AMI to the LF Account
  3. LF account shares AMI to Meta account and Allow Meta runners to deploy the Windows AMI to LF Account

I think all of these options will work however option 2 is probably the simplest solution since Meta runners are already setup to work on pytorch/test-infra and would only require the sharing of AMI to the LF Account.

If we agree to go with option 2 then I think I can start working on this task to create a job to build the AMI in Meta's AWS Account and someone from Meta can setup the AMI sharing permissions with the LF account.

@atalman @ZainRizvi any opinions on which option you want to go with?

zxiiro added a commit to pytorch/ci-infra that referenced this issue Dec 13, 2024
This role allows Packer to run `packer build` to create AMIs.

Issue: pytorch/test-infra#5992
Signed-off-by: Thanh Ha <[email protected]>
@zxiiro
Copy link
Collaborator Author

zxiiro commented Dec 13, 2024

Okay so I realized I could create the job on pytorch/pytorch and have it fetch test-infra before running packer build. This will allow us to just build the AMI directly in the PyTorch AWS Account. I think we should build there so that it's done on the community side and we can share the AMI back to Meta's CI account.

This ci-infra PR pytorch/ci-infra#312 is ready to go. I've confirmed the permissions work to allow GHA to run packer build.

I'm currently testing pytorch/pytorch#143132 which adds the job in pytorch/pytorch and will update once I've confirmed it to be working and can successfully create the AMI.

zxiiro added a commit to pytorch/ci-infra that referenced this issue Dec 16, 2024
This role allows Packer to run `packer build` to create AMIs.

Issue: pytorch/test-infra#5992
Signed-off-by: Thanh Ha <[email protected]>
zxiiro added a commit to pytorch/ci-infra that referenced this issue Dec 16, 2024
This role allows Packer to run `packer build` to create AMIs.

Issue: pytorch/test-infra#5992
Signed-off-by: Thanh Ha <[email protected]>
zxiiro added a commit to pytorch/pytorch that referenced this issue Dec 16, 2024
Add a job to create Windows AMIs in the PyTorch AWS Account.

Issue: pytorch/test-infra#5992
Signed-off-by: Thanh Ha <[email protected]>
zxiiro added a commit to pytorch/pytorch that referenced this issue Dec 16, 2024
Add a job to create Windows AMIs in the PyTorch AWS Account.

Issue: pytorch/test-infra#5992
Signed-off-by: Thanh Ha <[email protected]>
zxiiro added a commit to pytorch/pytorch that referenced this issue Dec 16, 2024
Add a job to create Windows AMIs in the PyTorch AWS Account.

Issue: pytorch/test-infra#5992
Signed-off-by: Thanh Ha <[email protected]>
@zxiiro
Copy link
Collaborator Author

zxiiro commented Dec 16, 2024

I was able to create a job that uses ubuntu-latest GitHub hosted runner and no longer requires a self-hosted runner. We can likely deploy this directly in test-infra now. Will investigate this afternoon.

zxiiro added a commit that referenced this issue Dec 16, 2024
Add a job to create Windows AMIs in the PyTorch AWS Account.

Issue: #5992
Signed-off-by: Thanh Ha <[email protected]>
zxiiro added a commit that referenced this issue Dec 16, 2024
Add a job to create Windows AMIs in the PyTorch AWS Account.

Issue: #5992
Signed-off-by: Thanh Ha <[email protected]>
zxiiro added a commit that referenced this issue Dec 16, 2024
Add a job to create Windows AMIs in the PyTorch AWS Account.

Issue: #5992
Signed-off-by: Thanh Ha <[email protected]>
zxiiro added a commit that referenced this issue Dec 16, 2024
Add a job to create Windows AMIs in the PyTorch AWS Account.

Issue: #5992
Signed-off-by: Thanh Ha <[email protected]>
zxiiro added a commit to pytorch/ci-infra that referenced this issue Dec 16, 2024
Allow the pytorch/test-infra repo to be able to use the packer role.

Issue: pytorch/test-infra#5992
Signed-off-by: Thanh Ha <[email protected]>
zxiiro added a commit that referenced this issue Dec 16, 2024
Add a job to create Windows AMIs in the PyTorch AWS Account.

Issue: #5992
Signed-off-by: Thanh Ha <[email protected]>
zxiiro added a commit to pytorch/ci-infra that referenced this issue Dec 16, 2024
Allow the pytorch/test-infra repo to be able to use the packer role.

Issue: pytorch/test-infra#5992
Signed-off-by: Thanh Ha <[email protected]>
zxiiro added a commit to pytorch/ci-infra that referenced this issue Dec 16, 2024
Allow the pytorch/test-infra repo to be able to use the packer role.

Issue: pytorch/test-infra#5992
Signed-off-by: Thanh Ha <[email protected]>
atalman pushed a commit that referenced this issue Dec 17, 2024
Add a job to create Windows AMIs in the PyTorch AWS Account.

Issue: #5992

---------

Signed-off-by: Thanh Ha <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants