-
Notifications
You must be signed in to change notification settings - Fork 38
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
refactor: lockfile #95
Open
jjmaestro
wants to merge
15
commits into
GoogleContainerTools:main
Choose a base branch
from
jjmaestro:refactor-lockfile
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jjmaestro
force-pushed
the
refactor-lockfile
branch
2 times, most recently
from
September 22, 2024 18:43
a871658
to
ab87a69
Compare
jjmaestro
force-pushed
the
refactor-lockfile
branch
4 times, most recently
from
October 29, 2024 08:23
da44c65
to
90e74e3
Compare
jjmaestro
force-pushed
the
refactor-lockfile
branch
from
November 14, 2024 01:07
90e74e3
to
f47c059
Compare
jjmaestro
force-pushed
the
refactor-lockfile
branch
from
November 24, 2024 21:54
f47c059
to
bf7339a
Compare
This was referenced Nov 24, 2024
jjmaestro
force-pushed
the
refactor-lockfile
branch
from
November 28, 2024 12:47
bf7339a
to
4c76750
Compare
* Remove unnecessary state struct, just pass repository struct to the resolver methods * Remove unused resolve_package from the public API
While working with some flaky mirrors and trying to figure out why they were failing I found the _fetch_package_index code a bit hard to follow so here's my attempt at streamlining it a bit: * Change the general flow of the for-loop so that we can directly set the reasons for failure in failed_attempts * Remove both integrity as an argument and as a return value since neither is ever used. * return the content of the Packages index instead of the path so that (1) we don't need rctx anywhere else and (2) is easier to mock. * Reword failure messages adding more context and debug information * Shorter lines and templated strings, trying to make the code easier to read and follow.
Refactor _version_relop into a compare method in version.bzl plus a VERSION_OPERATORS dict so that (1) we use the operator strings everywhere and (2) we can use the keys to validate the operators.
`version_constraint.bzl` is a bunch of utils for package version dependency. Either it has enough entity to stand by itself or the functionality should be folded into `version.bzl` and `package_index.bzl` and/or `package_resolution.bzl`.
Tests should match the file they are testing
Cleanup the version constraint parsing using the VERSION_OPERATORS that we now have in the version struct plus also validating the version being parsed.
* Add assertions for unresolved_deps in apt_dep_resolver_test * rename resolve_all to resolve * Reorder (name, version, arch) args to match the order of the index keys (arch, name, version) * Break up the long lines in comments * Improve _ITERATION_MAX_ fail message * Remove the "# buildifier: disable=print" since it's no longer needed.
* Refactor the nested dict get-set logic into its own nested_dict struct * Remove all the (now) unnecessary wrappers like _package_versions, _virtual_packages, etc. * Fix get() method to return default_value if no keys are given * Add add() to nested dict struct so that we can simplify the virtual package logic in apt_deb_repository.bzl _add_package
Add a parse_provides method wich parses and validates 'Provides'
It's clear that the inside of the loop will never return a package if there's no version provided so we can move the `if version` from within the loop all the way out and greatly simplify the inner logic.
* Add testing for apt_deb_repository mocking the external / side effects (downloads, decompression, etc). * Rename _parse_repository back to _parse_package_index * Add test_util.asserts_dict_equals
Cleanup apt_dep_resolver_test removing the apt_deb_repository mock, it's not really needed once we have proper testing and mocking of apt_deb_repository that we can reuse.
* Refactor lockfile into v2 and add tests. The v2 lockfile format: * uses the nested_dict struct to store the packages so it doesn't need the fast_package_lookup dict. * has the dependencies sorted so the lockfile now has stable serialization and the diffs of the lock are actually usable and useful to compare with the changes to the manifest. * removes the package and dependency key from the lockfile and moves it to an external function in deb_import.bzl (make_deb_import_key). * Remove add_package_dependency from the lockfile API. Now, the package dependencies are passed as an argument to add_package. This way, the lockfile functionality is fully contained in lockfile.bzl and e.g. we can remove the "consistency checks" that were only needed because users could forget to add the dependency as a package to the lockfile. * Ensure backwards-compatibility by internally converting lock v1 to v2. Also, when a lock is set and it's in v1 format, there's a reminder that encourages the users to run @repo//:lock to update the lockfile format. * Move all of the "package logic" to pkg.bzl, including the package key (now removed from the index). * Add tests for pkg.bzl * Add mock_value struct to mocks to organize the large mock values used for testing lockfile, etc. * Finally, deb_import_key is centralized in deb_import and uses pkg.key
By separating the migration from the previous commit we get to 1. in the previous commit, run all tests with the new code while locks are still v1 2. update the locks n this commit to V2 so we can then re-run all tests in the final state.
jjmaestro
force-pushed
the
refactor-lockfile
branch
from
December 11, 2024 20:06
4c76750
to
ad1b8ca
Compare
I've rebased the PR again, no changes from before apart from the rebase. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note
Stacked on top of #93 (I moved this PR down in the stack)
refactor: lockfile, new
pkg.bzl
, and lock v1-to-v2 auto-migrationRefactor lockfile into v2 and add tests. The v2 lockfile format:
nested_dict
struct
to store the packages so it doesn't need the fast_package_lookup dict.make_deb_import_key
indeb_import.bzl
)Remove
add_package_dependency
from the lockfile API. Now, the package dependencies are passed as an argument to add_package. This way, the lockfile functionality is fully contained inlockfile.bzl
and e.g. we can remove the "consistency checks" that were only needed because users could forget to add the dependency as a package to the lockfile.Ensure backwards-compatibility by internally converting lock v1 to v2. Also, when a lock is set and it's in v1 format, there's a reminder that encourages the users to run @repo//:lock to update the lockfile format.
Move all of the "package logic" to
pkg.bzl
Add tests for
pkg.bzl
Add
mock_value
struct
tomocks.bzl
to organize the large mock values used for testinglockfile
, etc.Finally, deb_import_key is centralized in deb_import and uses pkg.key
chore: migrate repo locks to v2
By separating the migration from the previous commit we make sure that the change is backwards compatible: