Add Submodule Caching to Toolchain Generation #1607
Conversation
|
Workflow can be seen here |
TShapinsky
force-pushed
the
cache-submodules
branch
6 times, most recently
from
9fe9342 to
3862f0e
Compare
TShapinsky
force-pushed
the
cache-submodules
branch
4 times, most recently
from
9351157 to
9f469aa
Compare
.github/workflows/build.yaml
Outdated
| id: submodule-hash | ||
| run: echo "HASH=$(git submodule | sha1sum | head -c 40)" >> $GITHUB_OUTPUT | ||
|
|
||
| - name: Check is Cache Exists for Exact Submodule Configuration |
There was a problem hiding this comment.
Typo? "is" should be "if"?
Very minor cosmetic (pedantic?!) issue - why the capitalisation of words in comments (here and elsewhere)?
There was a problem hiding this comment.
Thanks for catching the typo. I was using title case because they appear in the CI as the titles of each step. But if the standard here is all lower case or sentence case I'll happily switch to that.
There was a problem hiding this comment.
Thanks for catching the typo. I was using title case because they appear in the CI as the titles of each step. But if the standard here is all lower case or sentence case I'll happily switch to that.
Ah - they're names of the steps/tasks. I mistakenly thought that they were comments. If that's the convention then it makes sense to adhere to it. But should every word start with a capital letter? Not all do - e.g. "Check if Cache Exists for Exact Submodule Configuration", "If no Cache Hit, Update Cache", "Clone submodules", etc. Maybe prepositions are exempt from the rule? :-)
There was a problem hiding this comment.
Yeah, maybe just the first word should be capital. Title case is what I was going for, link, which does not capitalize prepositions or conjunctions.
TShapinsky
force-pushed
the
cache-submodules
branch
2 times, most recently
from
3b609a8 to
06a5cf2
Compare
TShapinsky
force-pushed
the
cache-submodules
branch
from
06a5cf2 to
d620298
Compare
Adds a job at the beginning of the build workflow which updates caches containing the different submodules required to build the different variants.
This job takes about 3 minutes to run the first time, and after that it takes about 4 seconds to check that the cache is still up to date. This could also be moved to a different trigger which runs less often.
This greatly reduces the amount of resources used to clone submodules (just for the build variants for now). The clones that do happen are shallow (except for uclibc-ng) which also shrinks the git overhead by at least 1GB.
This action only caches the
.git/modulesdirectory as the rest of the working directory can be rebuilt from those without needing to download anything else. This allows us to choose which submodules we want to manifest into our build environment. The directory sizes for GCC and LLVM are 1.1GB and 1.9GB respectively, so being able to pick which we want in our environment saves a lot of disk space.An additional note on github caches. They are scoped to branches, but any branch can restore a cache that was saved by
master. Repos are limited to 10GB of total cache storage, so restricting cache generation tomastermay also be a good move to prevent using all of the cache storage capacity.