Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pkg/cache: support creating a json cache directly from a model.Model #1111

Closed

Conversation

joelanford
Copy link
Member

@joelanford joelanford commented Jun 13, 2023

Description of the change:
Add cache.LoadJSONFromModel

Motivation for the change:
We removed functionality during the cache refactor that made it impossible to build a GRPCQuery implementation directly from a model.Model.

This PR adds the ability to so this, specifically in the cache.JSON implementation.

I'm not convinced this is the best approach. Another approach may be to introduce a new (optional?) interface method, something like BuildWithModel that a cache implementation could support. This approach would require some thinking about how cache.New would work though. Because it returns a "preferred" cache implementation if it detects an empty cacheDir. What if the preferred cache implementation doesn't support BuildWithModel?

Closes #1116

Reviewer Checklist

  • Implementation matches the proposed design, or proposal is updated to match implementation
  • Sufficient unit test coverage
  • Sufficient end-to-end test coverage
  • Docs updated or added to /docs
  • Commit messages sensible and descriptive

@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 13, 2023
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jun 13, 2023

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jun 13, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: joelanford

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 13, 2023
Comment on lines +212 to +214
if err := os.WriteFile(filepath.Join(q.baseDir, jsonDigestFile), []byte(digest), jsonCacheModeFile); err != nil {
return err
}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should create an empty digest file in the case that we're loading from a model?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this be done in LoadJSONFromModel?

Would you ever do a integrity check when you're loaded directly from a model?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean from my perspective I'd call LoadJSONFromModel with a temporary directory and throw the directory away when I am done.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you ever do a integrity check when you're loaded directly from a model?

I don't think so. The digest is a combination of the filesystem contents of the FBC and the cache, so if either change, then the cache is invalid. In this case, we have no FBC filesystem, so the integrity check doesn't make sense.

I mean from my perspective I'd call LoadJSONFromModel with a temporary directory and throw the directory away when I am done.

Yes, that would be the expected use of this function. But I'm not happy with the API because it never makes sense to do anything other than that. In which case, why ask the caller to pass a directory in? Ideally, the temporary directory would be an implementation detail not exposed in the API. But then how does the caller clean up when they're done? The Cache API design expects the cache to be re-usable and saved for later, so there is no Close() or Cleanup() method in the cache API or in the JSON cache struct.

Maybe need a higher level abstraction that uses a cache under the hood?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, the "magic" happens in the packagesFromModel function so maybe the easier approach is to expose that publicly and then call the functions on the packageIndex? This seems to sidestep the caching mechanism entirely if I am not mistaken.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some of those functions accept a cache though (e.g.

func (pkgs packageIndex) GetBundleForChannel(ctx context.Context, c Cache, pkgName string, channelName string) (*api.Bundle, error) {
)

So right now, packageIndex is very much coupled to the cache.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah.. I see that now. My mistake.

@codecov
Copy link

codecov bot commented Jun 13, 2023

Codecov Report

Merging #1111 (3c775b8) into master (b51aaf0) will increase coverage by 0.24%.
The diff coverage is 68.00%.

❗ Current head 3c775b8 differs from pull request most recent head 33ec102. Consider uploading reports for the commit 33ec102 to get more accurate results

@@            Coverage Diff             @@
##           master    #1111      +/-   ##
==========================================
+ Coverage   52.35%   52.59%   +0.24%     
==========================================
  Files         107      108       +1     
  Lines        9688     9712      +24     
==========================================
+ Hits         5072     5108      +36     
+ Misses       3674     3661      -13     
- Partials      942      943       +1     
Impacted Files Coverage Δ
pkg/cache/cache.go 32.35% <0.00%> (ø)
pkg/cache/json.go 50.00% <20.00%> (-3.16%) ⬇️
alpha/declcfg/load.go 74.05% <62.50%> (-1.28%) ⬇️
alpha/template/composite/builder.go 83.44% <80.00%> (ø)
alpha/declcfg/errors.go 91.89% <91.89%> (ø)
alpha/action/render.go 65.02% <100.00%> (ø)
alpha/declcfg/declcfg.go 78.94% <100.00%> (+32.07%) ⬆️
alpha/template/composite/composite.go 78.70% <100.00%> (ø)

... and 2 files with indirect coverage changes

@openshift-merge-robot openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 9, 2023
@openshift-merge-robot
Copy link
Contributor

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@joelanford joelanford closed this Nov 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Provide a replacement for the old querier mechanism
3 participants