Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the native ARM64 toolchain build to the main CI #847

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

hjyamauchi
Copy link
Collaborator

  • Introduce the concept of ARM64 CI runners. Use the cirun ARM64 machines as the ARM64 runners.
  • Split the windows_build job into X64 and ARM64.
  • Add the Win/ARM64 build on the Win/ARM64 runner (but no Win/X64 build on Win/ARM64 runner) while retaining the existing Win/X64 and Win/ARM64 builds on the Win/X64 runner.
  • Split some of strategy matrices between X64 and ARM64.
  • Append the "... (building on )" to the job names to clarify what's building on what runner.
  • Do not release the natively-released Win/ARM64 installer.exe while retaining the existing cross-compiled Win/ARM64 and Win/X64 installer.exe as the published release.
  • Exclude the Android-related components including ds2 in the native Win/ARM64 build due to the non-existence of the native Windows/ARM64 Android NDK.
  • Other misc changes to get the jobs to be able to run natively on Win/ARM64 (e.g., overriding the build/host arch to arm64 for the vsdevenv or the swift toolchain setup actions).
  • The native Win/ARM64 build should currently fail in the compilers job due to the static_assert issue in the pinned toolchain but the whole main CI run should succeed and produce builds as the release job depends only on windows-build-x64.

.github/workflows/build-toolchain.yml Outdated Show resolved Hide resolved
.github/workflows/build-toolchain.yml Outdated Show resolved Hide resolved
.github/workflows/build-toolchain.yml Outdated Show resolved Hide resolved
path: ${{ github.workspace }}/tmp/amd64

- uses: actions/download-artifact@v4
with:
name: installer-arm64
name: installer-arm64-built-on-amd64
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We shouldn't be changing the name of the artifact, the native vs cross should generate the same content.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem is that the upload-artifact fails and causes the entire job to fail if something with the same name has already been uploaded and the order between x64/arm64, which depends on the build progress speed, is unpredictable. Can you think of a better workaround?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That shouldn't matter - we can download only the native builds (i.e. the amd64 installer from x64, the arm64 installer from arm64).

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wdym? The issue is that an artifact name conflict will cause an error in the CI run. The way it's set up in this PR, both amd64/arm64 release installers come from the amd64 build and the arm64 build is only building to check the buildability.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right; I'm saying we change that. Only the AMD64 build comes from the AMD64 build and only the ARM64 build comes from the ARM64 build.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the main issue is that you are building the same artifacts with both an arm64 and an x64 host (see my other comment). Basically, only one host should be building for one given target. Anything more is redundant.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As above and what compnerd and I discussed in other places, the premise is that we want to keep three builds: x64-built-on-x64, arm64-built-on-x64, arm64-built-on-arm64. They are redundant from the point of view of producing release binaries but they are not from the point of view of ensuring the three builds work.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we want to keep maintaining the arm64-built-on-x64 build then yes, we don't have a choice, we need the ability to differentiate between the 2 builds or we are going to hit a conflict with the artifact name.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gotcha

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@compnerd Are you okay with this suffix approach?

@@ -662,17 +827,17 @@ jobs:

release:
runs-on: ubuntu-latest
needs: [context, windows-build]
needs: [context, windows-build-x64]
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be dependent on both of the architecture builds.

Copy link
Collaborator Author

@hjyamauchi hjyamauchi Nov 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. windows-build-arm64 currently fails. Do you know of a way to make it continue on an arm64 failure?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can mark a job as being ignored IIRC, I don't remember the syntax off hand, but the GHA documentation includes it.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added continue-on-error: true to windows-build.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Used if success() || failure() on release instead of continue-on-error as it didn't work.

@hjyamauchi
Copy link
Collaborator Author

hjyamauchi commented Nov 20, 2024

@compnerd PTAL

Copy link
Owner

@compnerd compnerd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that the native only build usage would greatly simplify this change.

.github/workflows/build-toolchain.yml Outdated Show resolved Hide resolved
.github/workflows/build-toolchain.yml Outdated Show resolved Hide resolved
path: ${{ github.workspace }}/tmp/amd64

- uses: actions/download-artifact@v4
with:
name: installer-arm64
name: installer-arm64-built-on-amd64
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That shouldn't matter - we can download only the native builds (i.e. the amd64 installer from x64, the arm64 installer from arm64).

@hjyamauchi
Copy link
Collaborator Author

hjyamauchi commented Nov 20, 2024

I think that the native only build usage would greatly simplify this change.

What do you mean by "the native only build usage" exactly?

.github/workflows/build-toolchain.yml Outdated Show resolved Hide resolved
path: ${{ github.workspace }}/tmp/amd64

- uses: actions/download-artifact@v4
with:
name: installer-arm64
name: installer-arm64-built-on-amd64
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the main issue is that you are building the same artifacts with both an arm64 and an x64 host (see my other comment). Basically, only one host should be building for one given target. Anything more is redundant.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants