Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: re-organize GitHub Actions #4600

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

liushuyu
Copy link
Contributor

This pull request re-organizes GitHub Actions, where Linux and macOS tasks are now split.
This is needed because the Linux build environment uses Docker containers to minimize the impact of GitHub Actions adding/removing PPAs from their host VM systems.

In addition, Linux build containers switch to LLVM's upstream APT archive instead of downloading the binary files from GitHub Release.

A LLVM 18.1 build job is also added to the build matrix to support #4599.

@liushuyu liushuyu force-pushed the new-ci-1 branch 2 times, most recently from 75e88c6 to 991baf3 Compare March 23, 2024 05:19
@kinke
Copy link
Member

kinke commented Mar 23, 2024

This is needed because the Linux build environment uses Docker containers to minimize the impact of GitHub Actions adding/removing PPAs from their host VM systems.

Which is a problem because of the LLVM apt repo now? - We've used that apt repo in the past, years ago, and it wasn't reliable IIRC - downtimes, as well as conflicts with the CI runner image from time to time [with whatever CI system we had back then, way before GitHub Actions came around].

The availability of binary LLVM packages from GitHub directly has always been a bit problematic, especially since we cannot use Ubuntu 22 due to debuginfo/gdb regressions (#4389).

@liushuyu
Copy link
Contributor Author

Which is a problem because of the LLVM apt repo now? - We've used that apt repo in the past, years ago, and it wasn't reliable IIRC - downtimes, as well as conflicts with the CI runner image from time to time [with whatever CI system we had back then, way before GitHub Actions came around].

We can instead make a prebuilt Docker container image so that apt.llvm.org availability problem will be avoided. Since it's inside a container, it won't be influenced by the host system and can be easily migrated if another CI system is used in the future.

The availability of binary LLVM packages from GitHub directly has always been a bit problematic, especially since we cannot use Ubuntu 22 due to debuginfo/gdb regressions (#4389).

Has the issue been forwarded to the GDB upstream? If yes, was this issue fixed in a newer GDB release?

@kinke
Copy link
Member

kinke commented Mar 24, 2024

We can instead make a prebuilt Docker container image so that apt.llvm.org availability problem will be avoided.

I think that'd be a bit too much effort. :) - What we have usually done so far is wait a bit for the new LLVM major to mature; then almost every time around there was at least one usable GitHub artifact among the various patch releases. For the recent LLVM versions without Ubuntu 20 packages, we've used macOS arm64 ones as you've seen. I expect one to pop up soon for v18.1.x too.

On rare occasions, we've been temporarily resorting to our LLVM fork and its artifacts. Those are normally only used for the main.yml GHA workflow and the LDC CI/release artifacts. The first prebuilt packages based on v18.1.* are available here (optionally with enabled assertions): https://github.com/ldc-developers/llvm-project/releases/tag/CI

The LLVM apt repo dependency aside, I'm not against a container-based approach in principle, but I definitely don't like the current duplication by splitting up the existing job into 2. There's probably a way to set something like container: ${{ null }} for the Mac jobs, to keep a single job definition. Possibly of interest: https://stackoverflow.com/questions/76925314/conditional-use-of-container-depending-on-the-group-it-runs-on-for-github-action

Has the issue been forwarded to the GDB upstream? If yes, was this issue fixed in a newer GDB release?

Not that I know of, mainly because it's not clear whether it isn't our fault. The same tests seem to pass with the DMD compiler on Ubuntu 22, using its default gdb version.

@liushuyu
Copy link
Contributor Author

I think that'd be a bit too much effort. :) - What we have usually done so far is wait a bit for the new LLVM major to mature; then almost every time around there was at least one usable GitHub artifact among the various patch releases. For the recent LLVM versions without Ubuntu 20 packages, we've used macOS arm64 ones as you've seen. I expect one to pop up soon for v18.1.x too.

I have already prepared a possible solution for making container images: https://github.com/liushuyu/ldc-build-environments.

The LLVM apt repo dependency aside, I'm not against a container-based approach in principle, but I definitely don't like the current duplication by splitting up the existing job into 2. There's probably a way to set something like container: ${{ null }} for the Mac jobs, to keep a single job definition. Possibly of interest: https://stackoverflow.com/questions/76925314/conditional-use-of-container-depending-on-the-group-it-runs-on-for-github-action

If using prebuilt container images, we can merge the tasks back and skip all the setup steps for Linux.

@liushuyu liushuyu force-pushed the new-ci-1 branch 4 times, most recently from 240389d to bf48024 Compare March 30, 2024 20:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants