Feature: Create an interface for downstream CIP integrations. #480

mattmoor · 2023-01-02T23:28:42Z

🎁 This change factors a new small library ./pkg/policy which is intended to streamline incorporating CIP validation into downstream tooling.

For a (much) more verbose explanation see here, but the general idea behind this is to allow CIP's to gate consumption of images in other contexts, for example the base images in build tools such as ko or kaniko. The idea is to enable the tool providers to bake-in default policies for default base images, and optionally expose configuration to let users write policies to authorize base images prior to consumption.

For example, I might write the following .ko.yaml:

verification:
  noMatchPolicy: deny
  policies:
  - data: |
      # inline policy
  - url: https://github.com/foo/bar/blobs/main/POLICY.yaml

With this library, it is likely <100 LoC to add base image policy verification to ko, and significantly simplifies our own policy-tester which has spaghetti code replicating some of this functionality.

/kind feature

Signed-off-by: Matt Moore [email protected]

Release Note

This adds a new ./pkg/policy that can be used to integrate policy verification into additional tooling.

Documentation

N/A

mattmoor · 2023-01-02T23:30:12Z

Staging this mostly for CI for now, I want to knock off the following before this lands:

Fold the path/url handling from policy-tester into ./pkg/policy,
Add ./pkg/policy/README.md documenting how folks should integrate this.

(I'll keep updating this comment as more things come to mind)

codecov-commenter · 2023-01-02T23:30:44Z

Codecov Report

Merging #480 (07d59dc) into main (d6ef1f3) will increase coverage by 1.40%.
The diff coverage is 76.97%.

❗ Current head 07d59dc differs from pull request most recent head 766d824. Consider uploading reports for the commit 766d824 to get more accurate results

@@            Coverage Diff             @@
##             main     #480      +/-   ##
==========================================
+ Coverage   54.73%   56.13%   +1.40%     
==========================================
  Files          38       42       +4     
  Lines        4129     4407     +278     
==========================================
+ Hits         2260     2474     +214     
- Misses       1688     1737      +49     
- Partials      181      196      +15

Impacted Files	Coverage Δ
pkg/policy/parse.go	`61.53% <61.53%> (ø)`
pkg/policy/verifier.go	`68.08% <68.08%> (ø)`
pkg/policy/policy.go	`91.54% <91.54%> (ø)`
pkg/policy/validate.go	`93.75% <93.75%> (ø)`

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

🎁 This change factors a new small library `./pkg/policy` which is intended to streamline incorporating CIP validation into downstream tooling. For a (much) more verbose explanation see [here](ko-build/ko#356 (comment)), but the general idea behind this is to allow CIP's to gate consumption of images in other contexts, for example the base images in build tools such as `ko` or `kaniko`. The idea is to enable the tool providers to bake-in default policies for default base images, and optionally expose configuration to let users write policies to authorize base images prior to consumption. For example, I might write the following `.ko.yaml`: ```yaml verification: noMatchPolicy: deny policies: - data: | # inline policy - url: https://github.com/foo/bar/blobs/main/POLICY.yaml ``` With this library, it is likely <100 LoC to add base image policy verification to `ko`, and significantly simplifies our own `policy-tester` which has spaghetti code replicating some of this functionality. /kind feature Signed-off-by: Matt Moore <[email protected]>

mattmoor · 2023-01-03T02:48:01Z

For context, this is motivated to support efforts like ko-build/ko#918 🤞

mattmoor · 2023-01-03T02:48:36Z

cc @vaikas @hectorj2f

vaikas

Sweet!

vaikas · 2023-01-03T09:34:25Z

pkg/policy/verifier.go

+		switch i.verification.NoMatchPolicy {
+		case "allow":
+			return nil
+		case "warn":


Since this is a library, I'm wondering if it would be better to wrap this as a warning "error type' (like it's done by the functions that we call below) and return it instead of having a separate function that one has to implement to see the warnings. Reasoning being that it would be easier for the implementor to control how to handle warnings and act on them. Looking at the warningwriter to see errors does not seem that friendly. Here and below. I guess I'm thinking that it would be useful to act in some cases on the fact that it's a warning, and we have the information already.

Generally because most Go code treats normal return values and errors as mutually exclusive, so carrying warnings as wrapped errors when also continuing with normal execution is sort of awkward. Originally I just logged it, but that presents its own "fun" (I actually need to tweak some of our logging for this to be usable, since as-is some paths spew zap logs 😞 ).

With the apis.Validatable pattern, the use of wrapped errors was pragmatic because:

It only returns an error, so it doesn't have the "other result" problem,

It only deals with *apis.FieldError, so we could add methods to mix/separate warnings/errors,

This kept us from breaking a several years old API! 😅

I opted for this because you can trivially accumulate warnings with a little function like:

func(s string, i ...interface{}) { warns = append(warns, fmt.Sprintf(s, i...)) }

... and then we don't have to worry about plumbing anything via return values that virtually every Go programmer will hold wrong. 😅 You can also log it with log.Printf or treat them as errors with log.Fatalf (I think I use t.Errorf in one of the tests 🤓 ).

Happy to discuss more. Sorry for the wall of text, just trying to dump some of my thinking here as I go through the comments.

Makes sense, in other places here, you have a return signature splitting warn/errs, so maybe we could change the return signature to return:
warnings, errors

But, this is not something I'm feeling super strong about, just was thinking from the perspective of the user that wants to easily distinguish if there are errors (already handled), or just warnings. Easy part was not intuitive given the signature of the warning handler function since it only took an interface :)

pkg/policy/verifier.go

vaikas · 2023-01-03T09:49:28Z

pkg/policy/validate.go

+				return
+			}
+
+		case corev1.SchemeGroupVersion.WithKind("Secret"):


This seems odd to me. We don't do anything with the Secret so validating (or even knowing about) it seems odd. Why are we doing this?

I think it would make reasoning about this Validate function clearer. That way the comment could read: "containing zero or more CIPs and validating them".

If the idea is that you can just cat a directory containing whatever into a single yaml, I guess I could see it, but that's not case, since if there's an unknown type we err, so yeah, just curious why the secret is special here.

Yeah, so this is here because I was lazy and just open sourced some of our own functions for parsing and validating some of this stuff, since I was just going to rewrite most of it anyways.

The Secret handling is currently unused here, but I thought it might be useful in a follow-up to support simulating secret references support in CIPs, WDYT? I can remove it, or open a tracking issue and //TODO(#1234) it, just lmk which you prefer.

I have a mild preference for the latter, since then we can just delete the code I open sourced once we pull this in, but it isn't that much code to maintain either way.

I am mainly thinking how this will start to look like assuming we start adding more things: configmaps, trustroots into the mix. That's going to make it more complicated to do more meaningful validation without starting to bleed more complexity here that will need to also be then replicated into the Compile. Secret reference support is actually a great example of that. Would the Validate unmarshal the key and make sure it's of the right type to make sure it's ok? Only to just to throw it away and redo that work in the Compile. As it stands now it doesn't do much as a validator, and actually could fail if the policy-controller is installed into a different namespace.

Not trying to be difficult here, just trying to visualize and see how this will work with more complex cases and what the division of labor is between Validate and Compile, and if Validate actually provides enough value besides making sure that the YAML can be semantically turned into objects we care about, does that make sense?

I think for CIPs it makes sense to do the Validate but for others, I am not quite convinced. At the very least, I think I'd remove the hardcoded cosign-system requirement.

Could you open a tracking issue for how much validation we really want to do besides for objects that we do not "own" (like secrets, configmaps) here vs. in Compile so we can maybe discuss it there.

pkg/policy/policy.go

pkg/policy/validate.go

pkg/policy/policy.go

pkg/policy/verifier.go

vaikas · 2023-01-03T10:33:02Z

pkg/policy/verifier.go

+			return nil, fmt.Errorf("unsupported policy shape: %v", p)
+		}
+
+		l, warns, err := ParseClusterImagePolicies(ctx, content)


Yeah, this is kind of what I was talking about in the 'Validate' function :) But I'm wondering why we parse them above and throw away the results (in the Validate function). Given the comment below on L107 about compilation, wondering if it would make sense to to rename Validate to Compile and have it return the parsed/fetched policies instead of redoing that work here again, and any changes have to be put in both places.

Or put another way, wonder if there's any reason to separate Validate / Compile into separate exposed functions, or just drop Validate and have Compile alone. Just wondering since Validate is public method, it's a bit confusing since Compile already calls it and does double the work right after calling Validate. The only thing it seems to do in addition is to convert to evaluatable policies. Also since Validate does not check for duplicate policies, seems better validation to just call Compile.

There wasn't as much duplication when I originally went with this shape because I only had Data! :)

Check out the way I refactored this in the next push (tl;dr private fetch method to share the logic to produce (string, error)).

I'd sort of like to keep the Validate function to drop our original version of the same method downstream (as I mentioned in the Secret comment, but as I said there, I don't care that much, just a nice to have).

pkg/policy/verifier.go

Signed-off-by: Matt Moore <[email protected]>

mattmoor · 2023-01-03T19:29:12Z

Failure looks like a flake, but I can't rerun it 😞

vaikas

I reran the tests and indeed it was a flake 😠 things pass now.

I'm going to merge it for now, and we can deal with the cleanups later.

vaikas · 2023-01-04T07:40:41Z

pkg/policy/validate.go

+				return
+			}
+
+		case corev1.SchemeGroupVersion.WithKind("Secret"):


I am mainly thinking how this will start to look like assuming we start adding more things: configmaps, trustroots into the mix. That's going to make it more complicated to do more meaningful validation without starting to bleed more complexity here that will need to also be then replicated into the Compile. Secret reference support is actually a great example of that. Would the Validate unmarshal the key and make sure it's of the right type to make sure it's ok? Only to just to throw it away and redo that work in the Compile. As it stands now it doesn't do much as a validator, and actually could fail if the policy-controller is installed into a different namespace.

Not trying to be difficult here, just trying to visualize and see how this will work with more complex cases and what the division of labor is between Validate and Compile, and if Validate actually provides enough value besides making sure that the YAML can be semantically turned into objects we care about, does that make sense?

I think for CIPs it makes sense to do the Validate but for others, I am not quite convinced. At the very least, I think I'd remove the hardcoded cosign-system requirement.

Could you open a tracking issue for how much validation we really want to do besides for objects that we do not "own" (like secrets, configmaps) here vs. in Compile so we can maybe discuss it there.

vaikas · 2023-01-04T07:47:48Z

pkg/policy/validate_test.go

+  images:
+  - glob: '*'
+`,
+			wantErr: errors.New("missing field(s): spec.authorities"),


I'd prefer if we could reuse the YAMLs from the existing test/testdata for these tests rather than adding these here. But we can do this in a followup if you agree. They are used for validating the admission webhook, so would be nice to keep them in sync I think.
https://github.com/sigstore/policy-controller/tree/main/test/testdata/policy-controller
warn,invalid,valid, empty-authorities

Each of the warn, error has the expected failure baked into it, like so:
https://github.com/sigstore/policy-controller/blob/main/test/testdata/policy-controller/warn/v1beta1-missing-identity.yaml#L15

https://github.com/sigstore/policy-controller/blob/main/test/testdata/policy-controller/invalid/invalid-pubkey.yaml#L15

So thinking we would get more thorough testing and as we add more testdata files, we'll get those test cases for free.

mattmoor force-pushed the add-policy-sdk branch 4 times, most recently from e867625 to 602772e Compare January 3, 2023 02:38

mattmoor changed the title ~~[WIP] Feature: Create an interface for downstream CIP integrations.~~ Feature: Create an interface for downstream CIP integrations. Jan 3, 2023

mattmoor force-pushed the add-policy-sdk branch from 602772e to 214c2dd Compare January 3, 2023 02:45

mattmoor mentioned this pull request Jan 3, 2023

[WIP] Feature: Add support for verifying base image policies. ko-build/ko#918

Open

4 tasks

vaikas reviewed Jan 3, 2023

View reviewed changes

mattmoor force-pushed the add-policy-sdk branch from 07d59dc to d082473 Compare January 3, 2023 19:06

Incorporate review feedback from vaikas

766d824

Signed-off-by: Matt Moore <[email protected]>

mattmoor force-pushed the add-policy-sdk branch from d082473 to 766d824 Compare January 3, 2023 19:06

vaikas approved these changes Jan 4, 2023

View reviewed changes

vaikas merged commit 8c7328d into sigstore:main Jan 4, 2023

leongross mentioned this pull request Jan 13, 2023

ci: build reproducible container images with ko edgelesssys/constellation#871

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature: Create an interface for downstream CIP integrations. #480

Feature: Create an interface for downstream CIP integrations. #480

mattmoor commented Jan 2, 2023

mattmoor commented Jan 2, 2023 •

edited

Loading

codecov-commenter commented Jan 2, 2023 •

edited

Loading

mattmoor commented Jan 3, 2023

mattmoor commented Jan 3, 2023

vaikas left a comment

vaikas Jan 3, 2023

mattmoor Jan 3, 2023

vaikas Jan 3, 2023

vaikas Jan 3, 2023

vaikas Jan 3, 2023

vaikas Jan 3, 2023

mattmoor Jan 3, 2023

vaikas Jan 4, 2023

vaikas Jan 3, 2023

vaikas Jan 3, 2023

mattmoor Jan 3, 2023

mattmoor commented Jan 3, 2023

vaikas left a comment

vaikas Jan 4, 2023

vaikas Jan 4, 2023

Feature: Create an interface for downstream CIP integrations. #480

Feature: Create an interface for downstream CIP integrations. #480

Conversation

mattmoor commented Jan 2, 2023

Release Note

Documentation

mattmoor commented Jan 2, 2023 • edited Loading

codecov-commenter commented Jan 2, 2023 • edited Loading

Codecov Report

mattmoor commented Jan 3, 2023

mattmoor commented Jan 3, 2023

vaikas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattmoor commented Jan 3, 2023

vaikas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattmoor commented Jan 2, 2023 •

edited

Loading

codecov-commenter commented Jan 2, 2023 •

edited

Loading