Build GHC with cabal-install and a Makefile #3

hsyl20 · 2024-11-27T12:10:25Z

Build ghc-stage1
Generate valid non-cross stage1 settings (reusing stage0's settings for the most part)
Build ghc-stage2
For every host/arch we want to build a compiler for:
- generate valid stage2 settings for the target (using ghc-toolchain?)
- use ghc-stage2 with these settings to build root libraries
- use ghc-stage2 with these settings to build iserv
- (optional) use ghc-stage2 with these settings to cross-build a GHC
...

As noted in #25509, the `-this-package-name` must be passed for each package to ensure that GHC can response references to the packages' exposed modules via package-qualified imports. Fix this. Closes #25509.

The default value for base-unit-id is stored in the settings file. At install time, this can be set by using the BASE_UNIT_ID environment variable. At runtime, the value can be set by `-base-unit-id` flag. For whether all this is a good idea, see #25382 Fixes #25382

As #14497 showed black holes can appear inside large objects when we capture a computation and later blackhole it like we do for AP_STACK closures. Fixes #24791

This patch makes some minor improvements re nix-in-docker logic in the ci configuration: - Update `nixos/nix` to the latest version - Apply $CPUS to `cores`/`max-jobs` to avoid oversubscribing while allowing a reasonable degree of parallelism - Remove redundant `--extra-experimental-features nix-command` in later `nix shell` invocations, it's already configured in `/etc/nix/nix.conf`

This patch makes test-bootstrap related ci jobs only depend on hadrian-ghc-in-ghci job to finish, consistent with other jobs in the full-build stage generated by gen_ci.hs. This allows the jobs to be spawned earlier and improve overall pipeline parallelism.

This is never used for lexing / parsing. It is only used by `GHC.Parser.Header.getOptions`.

Simply uses the multiplicity as stored in the field. As I'm writing this commit, the only possible multiplicity is 1, but !13525 is changing this. It's actually easier to take !13525 into account. Fixes #25515.

Fixes #24660

This patch bumps macOS minimum SDK version to 11.0 for x86_64-darwin to align it with aarch64-darwin. This allows us to get rid of the horrible -Wl,-U,___darwin_check_fd_set_overflow hack, which is causing linker warnings and testsuite failures on macOS 15. Fixes #25504.

See this CLC proposal: - haskell/core-libraries-committee#289 and this CLC proposal for background: - haskell/core-libraries-committee#288 Metric Decrease: MultiLayerModulesTH_OneShot

…form With the Medium code model, the jump range of the generated jump instruction is larger than that of the Small code model. It's a temporary fix of the problem descriped in https://gitlab.haskell .org/ghc/ghc/-/issues/25495. This commit requires that the LLVM used contains the code of commit 9dd1d451d9719aa91b3bdd59c0c6679 83e1baf05, i.e., version 8.0 and later. Actually we should not rely on LLVM, so the only way to solve this problem is to implement the LoongArch backend. Add new type for codemodel

hsyl20 · 2024-11-27T12:26:12Z

Current status: it builds some ghc program in _build/stage0/bin/ghc. It seems to be linked with the wrong ghc-boot because it reports a GHC version of 9.8.2 (my bootstrap GHC).

hsyl20 · 2024-11-27T12:29:42Z

Ah, I need to pass some environment variable to ghc-boot's Setup.hs

When constant folding ensure the result is still within bounds for the given type by explicitly narrowing the results. Not doing so results in a lot of spurious assembler warnings especially when testing primops.

We verify that required flags (currently `--output` and `--triple`) are provided. The implementation is truly awful, but so is getopt. Begins to address #25500.

Currently the ExecPage facility has two users: * GHCi, for constructing info tables, and * the adjustor allocation path Despite neither of these have any spatial locality constraints ExecPage was using the linker's `mmapAnonForLinker`, which tries hard to ensure that mappings end up nearby the executable image. This makes adjustor allocation needlessly subject to fragmentation concerns. We now instead return less constrained mappings, improving the robustness of the mechanism. Addresses #25503.

These were incorrectly changed by the automated refactoring of the `ghc-internal` migration. Fixes #25521.

This change means that the Hadrian multi target will include exactprint. In particular, this means that HLS will work on exactprint inside the GHC tree.

Fixes #25428

angerman

Thank you @hsyl20 this looks like a great start!

angerman · 2024-11-28T04:44:42Z

Makefile

@@ -0,0 +1,67 @@
+HADRIAN_SETTINGS_STAGE0 := $(shell ghc --info | runghc GenSettings.hs ghc-boot)


Why is this called HADRIAN_SETTINGS

That's how the environment variable is called in ghc-boot's Setup.hs ¯_(ツ)_/¯

angerman · 2024-11-28T04:45:27Z

Makefile

+	## Substituting variables
+	cp _build/stage0/src/ghc-bin/ghc-bin.cabal{.in,}
+	cp _build/stage0/src/libraries/ghc/ghc.cabal{.in,}
+	cp _build/stage0/src/libraries/ghc/GHC/CmmToLlvm/Version/Bounds.hs{.in,}
+	cp _build/stage0/src/libraries/ghc-boot/ghc-boot.cabal{.in,}
+	cp _build/stage0/src/libraries/ghc-boot-th/ghc-boot-th.cabal{.in,}
+	cp _build/stage0/src/libraries/ghc-heap/ghc-heap.cabal{.in,}
+	cp _build/stage0/src/libraries/ghci/ghci.cabal{.in,}
+	cp _build/stage0/src/utils/ghc-pkg/ghc-pkg.cabal{.in,}
+
+	sed -i 's/@ProjectVersion@/9.13/' _build/stage0/src/ghc-bin/ghc-bin.cabal
+	sed -i 's/@ProjectVersionMunged@/9.13/' _build/stage0/src/ghc-bin/ghc-bin.cabal
+	sed -i 's/@ProjectVersion@/9.13/' _build/stage0/src/libraries/ghc/ghc.cabal
+	sed -i 's/@ProjectVersionMunged@/9.13/' _build/stage0/src/libraries/ghc/ghc.cabal
+	sed -i 's/@ProjectVersion@/9.13/' _build/stage0/src/libraries/ghc-boot/ghc-boot.cabal
+	sed -i 's/@ProjectVersionMunged@/9.13/' _build/stage0/src/libraries/ghc-boot/ghc-boot.cabal
+	sed -i 's/@ProjectVersion@/9.13/' _build/stage0/src/libraries/ghc-boot-th/ghc-boot-th.cabal
+	sed -i 's/@ProjectVersionMunged@/9.13/' _build/stage0/src/libraries/ghc-boot-th/ghc-boot-th.cabal
+	sed -i 's/@Suffix@//' _build/stage0/src/libraries/ghc-boot-th/ghc-boot-th.cabal
+	sed -i 's/@SourceRoot@/./' _build/stage0/src/libraries/ghc-boot-th/ghc-boot-th.cabal
+	sed -i 's/@ProjectVersion@/9.13/' _build/stage0/src/libraries/ghc-heap/ghc-heap.cabal
+	sed -i 's/@ProjectVersionMunged@/9.13/' _build/stage0/src/libraries/ghc-heap/ghc-heap.cabal
+	sed -i 's/@ProjectVersionForLib@/9.13/' _build/stage0/src/libraries/ghc-heap/ghc-heap.cabal
+	sed -i 's/@ProjectVersion@/9.13/' _build/stage0/src/libraries/ghci/ghci.cabal
+	sed -i 's/@ProjectVersionMunged@/9.13/' _build/stage0/src/libraries/ghci/ghci.cabal
+	sed -i 's/@ProjectVersionForLib@/9.13/' _build/stage0/src/libraries/ghci/ghci.cabal
+	sed -i 's/@ProjectVersion@/9.13/'       _build/stage0/src/utils/ghc-pkg/ghc-pkg.cabal
+	sed -i 's/@ProjectVersionMunged@/9.13/' _build/stage0/src/utils/ghc-pkg/ghc-pkg.cabal
+	sed -i 's/@ProjectVersionForLib@/9.13/' _build/stage0/src/utils/ghc-pkg/ghc-pkg.cabal
+
+	sed -i 's/@LlvmMinVersion@/13/' _build/stage0/src/libraries/ghc/GHC/CmmToLlvm/Version/Bounds.hs
+	sed -i 's/@LlvmMaxVersion@/20/' _build/stage0/src/libraries/ghc/GHC/CmmToLlvm/Version/Bounds.hs


We could just use a configure script for this right?

We could. But we moved away from it upstream.

angerman · 2024-11-28T04:46:35Z

cabal.project-stage0

+  ./_build/stage0/src/ghc-bin/
+  ./_build/stage0/src/libraries/ghc
+  ./_build/stage0/src/libraries/directory/
+  ./_build/stage0/src/libraries/file-io/
+  ./_build/stage0/src/libraries/filepath/
+  ./_build/stage0/src/libraries/ghc-platform/
+  ./_build/stage0/src/libraries/ghc-boot/
+  ./_build/stage0/src/libraries/ghc-boot-th/
+  ./_build/stage0/src/libraries/ghc-heap
+  ./_build/stage0/src/libraries/ghci
+  ./_build/stage0/src/libraries/os-string/
+  ./_build/stage0/src/libraries/process/
+  ./_build/stage0/src/libraries/semaphore-compat
+  ./_build/stage0/src/libraries/time
+  ./_build/stage0/src/libraries/unix/
+  ./_build/stage0/src/libraries/Win32/
+  ./_build/stage0/src/utils/ghc-pkg
+  ./_build/stage0/src/utils/hsc2hs
+  ./_build/stage0/src/utils/unlit
+  ./_build/stage0/src/utils/genprimopcode/
+  ./_build/stage0/src/utils/deriveConstants/
+


Why do we put them in build/stage0?

I don't want to modify the source tree directly, this way it's easier to nuke the _build directory and restart from a clean state.

I think we don't need to copy all source files across. It is also annoying to do it propely with Make.

cabal will use a separate build directory anyway, only some generated cabal files will end up in the source tree.

We need to perform some substitution/file generations more than once (for the different stages). I still think it's cleaner to avoid dirtying the source directory. I don't want to have to fix bugs because we've done something for stage0 and forgot to do it again differently for stage1.

hsyl20 · 2024-11-28T13:30:50Z

Now it seems like cabal doesn't like empty package databases:

Error: [Cabal-9076]
failed to parse output of 'ghc-pkg dump'

Edit: that was my mistake (a debug statement in ghc-pkg...). Now it works.

Earlier versions of `directory` are racy on Windows due to #24382. Also includes necessary Hadrian bootstrap plan bump. Fixes #24382.

GitLab recommends using `https://` to clone submodules and provides the `GIT_SUBMODULE_FORCE_HTTPS` variable to force this. Fixes #25528.

Makefile

andreabedini · 2024-11-29T05:19:10Z

Effectively we want to completely disable the solver. We can add
active-respositories: :none to cabal.project and remove the source constraints.

hsyl20 · 2024-11-29T09:49:53Z

Effectively we want to completely disable the solver. We can add active-respositories: :none to cabal.project and remove the source constraints.

I've disabled it for stage1 where we start fresh to build boot libraries.

1. Make `staticInitStat`, `staticDeclStat`, `allocUnboxedConStatic`, `allocateStaticList`, `jsStaticArg` local to modules. 2. Remove unused `hdRawStr`, `hdStrStr` from Haskell and JavaScript (`h$pstr`, `h$rstr`, `h$str`). 3. Introduce a special type `StaticAppKind` enumeration and `StaticApp` to represent boxed scalar static applications. Originally, StaticThunk supported to pass Maybe when it became Nothing for initializied thunks in an alternatie way but it is not used anymore.

…tf8`. It became possible due of introduction strings unfloating at Sinker pass (#13185). Earns few more bytes at optimizations.

Code analysis shown that such optimization would be possible out of the box if `cachedIdentForId` allowed to do that for Haskell `Id`s which are represented by few JavaScript `Ident`s. It is a usual for strings which are represented at JavaScript as a pair of 2 values: the string content and the offset where to start reading actual string from the full content. Usually offset is 0 but technically we need to allow such complex structures to be treated as "global". Enabling it there shown that `genToplevelRhs` and `globalOccs` had inaccuracies in their implementations: 1. `globalOccs` operated over JavaScript's `Ident`s but for complex structures it didn't pay attention to the fact that different Idents actually could be pointed to same Id. Now the algo is changed to calculate occurencies for Ids. 2. `genToplevelRhs` didn't assume that different Idents pointed to same Id can have mixed order of occurence. But actually the order is important. Strings are encoded into 2 variables where first is content and second is offset and their order are not interchangeable. It is fixed by regeneration Idents from collected Ids which is fine because all Idents generation is passed through the Cache and they are quasi-stable.

We had previously renamed this function for consistency, but that caused unnecessary breakage

Cabal shouldn't automatically try to set them. We set them explicitly.

bgamari and others added 13 commits November 25, 2024 03:55

hadrian-ghci-multi: Pass -this-package-name in unit response files

6e1fbda

As noted in #25509, the `-this-package-name` must be passed for each package to ensure that GHC can response references to the packages' exposed modules via package-qualified imports. Fix this. Closes #25509.

Refactoring: Use OnOff more consistently for Extension

a05e4a9

Compacting GC: Handle black holes in large objects.

7f90f31

As #14497 showed black holes can appear inside large objects when we capture a computation and later blackhole it like we do for AP_STACK closures. Fixes #24791

Refactoring: Remove pSupportedExts from ParserOpts

caaf538

This is never used for lexing / parsing. It is only used by `GHC.Parser.Header.getOptions`.

Add test for #25515

41f8365

Desugar record notation with correct multiplicities

9279619

Simply uses the multiplicity as stored in the field. As I'm writing this commit, the only possible multiplicity is 1, but !13525 is changing this. It's actually easier to take !13525 into account. Fixes #25515.

Clarify INLINE unfolding optimization docs.

fcc3ae6

Fixes #24660

ghc-experimental: expose GHC.RTS.Flags, GHC.Stats

53f978c

See this CLC proposal: - haskell/core-libraries-committee#289 and this CLC proposal for background: - haskell/core-libraries-committee#288 Metric Decrease: MultiLayerModulesTH_OneShot

AndreasPK and others added 7 commits November 27, 2024 11:40

Cmm constant folding: Narrow results to operations bitwidth.

df42ba1

When constant folding ensure the result is still within bounds for the given type by explicitly narrowing the results. Not doing so results in a lot of spurious assembler warnings especially when testing primops.

ghc-toolchain: Introduce basic flag validation

bf3db97

We verify that required flags (currently `--output` and `--triple`) are provided. The implementation is truly awful, but so is getopt. Begins to address #25500.

base: Fix incorrect mentions of GHC.Internal.Numeric

c3fc9b8

These were incorrectly changed by the automated refactoring of the `ghc-internal` migration. Fixes #25521.

Add checkExact to toolTargets

a362b94

This change means that the Hadrian multi target will include exactprint. In particular, this means that HLS will work on exactprint inside the GHC tree.

Add test for #25428

e6c957e

Don't bypass MonoLocalBind in empty patterns

52d97f4

Fixes #25428

angerman reviewed Nov 28, 2024

View reviewed changes

bgamari and others added 3 commits November 28, 2024 10:26

hadrian: Bump directory bound to >=1.3.9

7890f2d

Earlier versions of `directory` are racy on Windows due to #24382. Also includes necessary Hadrian bootstrap plan bump. Fixes #24382.

mention -Iw in +RTS -?

0fd43ea

gitlab-ci: Set GIT_SUBMODULE_FORCE_HTTPS

6cf579b

GitLab recommends using `https://` to clone submodules and provides the `GIT_SUBMODULE_FORCE_HTTPS` variable to force this. Fixes #25528.

andreabedini reviewed Nov 29, 2024

View reviewed changes

Makefile Outdated Show resolved Hide resolved

GulinSS and others added 22 commits December 8, 2024 13:52

JS: Add trivial optimizations for unpackCString and `unpackCStringU…

a9f8f1f

…tf8`. It became possible due of introduction strings unfloating at Sinker pass (#13185). Earns few more bytes at optimizations.

Fix panic in multiline string with unterminated gap (#25530)

a8ceccf

Add test case for unterminated multiline string

9e464ad

Revert mapMG renaming

ed1ed5c

We had previously renamed this function for consistency, but that caused unnecessary breakage

RTS: make Cabal flags manual

158261f

Cabal shouldn't automatically try to set them. We set them explicitly.

Add github workflow

342b6f9

Build GHC with cabal-install and a Makefile

12ce213

Generate proper settings for ghc-boot

a332a9f

Build stage0 tools: ghc-pkg, deriveConstants, genprimopcode

07c4028

Generate proper stage1 settings

7dd6f53

Fix installation of stage1 programs. WIP using stage1 compiler

2070d06

Fix path to stage1 package db

24dc759

Progress towards building the RTS

78bbedf

Fix CPP for Cmm

0a530e8

Apply suggestions

e3d5ac8

Allow specifying custom cabal

c62d560

Start with deriveConstants

2ddd93a

Generate DerivedConstants.h with a hack

0d60b8a

Pass -this-unit-id because cabal doesn't when it compiles Cmm files

9e62f97

Try using per source flags

563f2cf

hsyl20 force-pushed the wip/make-build branch from 4f9641d to 563f2cf Compare December 11, 2024 15:35

hsyl20 and others added 6 commits December 11, 2024 17:36

Fix rts per file flags

9dd5b6d

Generate AutoApply*.cmm

98032bd

Build libffi

e330ded

Progress building ghc-prim

7dbb512

Build ghc-prim

1174e85

refactor: avoid hack when checking size of struct MD5Context

1df74eb

andreabedini force-pushed the wip/make-build branch from 09d2c38 to 1df74eb Compare December 19, 2024 06:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build GHC with cabal-install and a Makefile #3

Build GHC with cabal-install and a Makefile #3

hsyl20 commented Nov 27, 2024 •

edited

Loading

hsyl20 commented Nov 27, 2024

hsyl20 commented Nov 27, 2024

angerman left a comment

angerman Nov 28, 2024

hsyl20 Nov 28, 2024

angerman Nov 28, 2024

hsyl20 Nov 28, 2024

angerman Nov 28, 2024

hsyl20 Nov 28, 2024

andreabedini Nov 29, 2024

hsyl20 Nov 29, 2024

hsyl20 commented Nov 28, 2024 •

edited

Loading

andreabedini commented Nov 29, 2024

hsyl20 commented Nov 29, 2024

		@@ -0,0 +1,67 @@
		HADRIAN_SETTINGS_STAGE0 := $(shell ghc --info \| runghc GenSettings.hs ghc-boot)

Build GHC with cabal-install and a Makefile #3

Are you sure you want to change the base?

Build GHC with cabal-install and a Makefile #3

Conversation

hsyl20 commented Nov 27, 2024 • edited Loading

hsyl20 commented Nov 27, 2024

hsyl20 commented Nov 27, 2024

angerman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hsyl20 commented Nov 28, 2024 • edited Loading

andreabedini commented Nov 29, 2024

hsyl20 commented Nov 29, 2024

hsyl20 commented Nov 27, 2024 •

edited

Loading

hsyl20 commented Nov 28, 2024 •

edited

Loading