build: init mimas #501
Merged
build: init mimas #501
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This comment was marked as off-topic.
This comment was marked as off-topic.
|
The script to migrate the state from rhea as found in #!/usr/bin/env bash
NIX_STORE=daemon
FROM=rhea.nixos.org
set -eux
mkdir -p /var/lib/hydra/queue-runner/keys/
mkdir -p /var/lib/hydra/www/keys/
mkdir -p /var/lib/hydra/{build-logs,runcommand-logs}
mkdir -p /nix/var/nix/gcroots/hydra
rsync -avPz "${FROM}":/etc/nix/machines /etc/nix/machines
rsync -avPz "${FROM}":/var/lib/hydra/queue-runner/keys/ /var/lib/hydra/queue-runner/keys
rsync -avPz "${FROM}":/var/lib/hydra/queue-runner/.aws/ /var/lib/hydra/queue-runner/.aws
rsync -avPz "${FROM}":/var/lib/hydra/queue-runner/.ssh/ /var/lib/hydra/queue-runner/.ssh
rsync -avPz "${FROM}":/var/lib/hydra/www/keys/ /var/lib/hydra/www/keys
rsync -avPz "${FROM}":/var/lib/hydra/build-logs/ /var/lib/hydra/build-logs || true
rsync -avPz "${FROM}":/var/lib/hydra/runcommand-logs/ /var/lib/hydra/runcommand-logs || true
rsync -avPz "${FROM}":/nix/var/nix/gcroots/hydra/ /nix/var/nix/gcroots/hydra || true
find /nix/var/nix/gcroots/hydra \( -type l -o -type f \) -print0 \
| while IFS= read -r -d $'\0' file; do
test -e "$file" || echo "/nix/store/$(basename "$file")"
done \
| xargs -P 16 nix-copy-closure --from --gzip --include-outputs --use-substitutes "${FROM}" |
|
Reformatted the NVME namespaces to use 4k blocks. nvme-SAMSUNG_MZQL21T9HCJR-00A07_S64GNNFX604905nvme-SAMSUNG_MZQL21T9HCJR-00A07_S64GNNFX604919 |
mweinelt
force-pushed
the
mimas
branch
5 times, most recently
from
ab028e1 to
dc5dcfe
Compare
Mic92
reviewed
Dec 6, 2024
|
Hydra is now running on mimas. I runtime-masked the hydra services on rhea for now. Will get some sleep and check back tomorrow. |
mweinelt
force-pushed
the
mimas
branch
2 times, most recently
from
f9fab7b to
0adfb7f
Compare
vcunat
reviewed
Dec 6, 2024
mweinelt
force-pushed
the
mimas
branch
2 times, most recently
from
1c88182 to
2c42161
Compare
We currently hold over 430GiB of build logs, which makes up 25% of the total disk space used on mimas.
Our hydra fork has a limit on the number of active build slots, which defaults to the number of threads minus two, so 96-2 = 94. vcunat looked at mimas graphs when we were hitting this limit, and it seemed OK to push more (negligible cpusome PSI).
|
Alright. There is much to improve still, but the migration looks pretty solid to me. Let's leave rhea around for another week just in case we've missed something and then cancel the contract. |
vcunat
reviewed
Dec 11, 2024
| @@ -93,6 +93,9 @@ in | |||
|
|
|||
| max_concurrent_evals = 1 | |||
|
|
|||
| # increase the number of active compress slots (CPU is 48*2 on mimas) | |||
| max_local_worker_threads = 144 | |||
There was a problem hiding this comment.
Now we had a longer period of "Copying Results" hovering roughly around 130, and PSI kept still very low. So I believe this certainly still isn't a too high value. We'll see over time. So far it's been relatively rare to get so high anyway.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Work in progress configuration for a successor to rhea, because the machine cannot keep up with the compression workload by the large number of builders we employ.
The plan is for an AX162-R at Hetzner.
TODO: