Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add tests for ListContainerStats #857

Merged
merged 2 commits into from
May 24, 2022

Conversation

minbaev
Copy link
Contributor

@minbaev minbaev commented Dec 13, 2021

Signed-off-by: Alexander Minbaev [email protected]

What type of PR is this?

/kind feature

What this PR does / why we need it:

PR adds tests for ListContainerStats RPC
Adds tests covering the RPC behavior after this change:
containerd/containerd#6373

Special notes for your reviewer:

None

Does this PR introduce a user-facing change?

None

@k8s-ci-robot k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Dec 13, 2021
@k8s-ci-robot
Copy link
Contributor

Welcome @minbaev!

It looks like this is your first PR to kubernetes-sigs/cri-tools 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/cri-tools has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Dec 13, 2021
@minbaev
Copy link
Contributor Author

minbaev commented Dec 13, 2021

/cc @mikebrow

@k8s-ci-robot
Copy link
Contributor

@minbaev: GitHub didn't allow me to request PR reviews from the following users: mikebrow.

Note that only kubernetes-sigs members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @mikebrow

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Copy link
Contributor

@mikebrow mikebrow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/LGTM with nits addressed after we have the fix in containerd merged into main, and cherry picked to containerd 1.5.9 should only have to "pause" for this test case for a short period. If it takes to long for the 1.5.9 release to cycle up over this holiday feel free to add a skip.

pkg/validate/container.go Outdated Show resolved Hide resolved
pkg/validate/container.go Outdated Show resolved Hide resolved
@feiskyer
Copy link
Member

could rebase again (it seems rebase is required to get new changes for github actions)?

@minbaev minbaev force-pushed the add-list-stat-test branch 2 times, most recently from 1abdac9 to 8aec4be Compare January 17, 2022 05:17
@minbaev
Copy link
Contributor Author

minbaev commented Jan 17, 2022

could rebase again (it seems rebase is required to get new changes for github actions)?

sure, done

@minbaev
Copy link
Contributor Author

minbaev commented Jan 18, 2022

@feiskyer @mikebrow
can you please review and see if it can be merged?
Thank you!

@saschagrunert
Copy link
Member

@minbaev the tests seems to fail, can we fix them?

@minbaev
Copy link
Contributor Author

minbaev commented Jan 19, 2022

@minbaev the tests seems to fail, can we fix them?

hmm, I don't think the tests are wrong, they pass when i run them locally against a local containerd instance with the change.

we cut a 1.5.9 containerd version which should include the change
containerd/containerd#6373

can it be possible that the tests fail due some CI issue?

Here is the output when I run the tests locally

root@ubuntu-bm:~/go/src/github.com/kubernetes-sigs/cri-tools# build/bin/critest -ginkgo.focus stats
critest version: 1.22.0-79-g8aec4be0
Running Suite: CRI validation
=============================
Random Seed: 1642586400 - Will randomize all specs
Will run 4 of 89 specs

Jan 19 04:00:00.504: INFO: Testing context container image list file:
Jan 19 04:00:00.504: INFO: Testing context container image list: {DefaultTestContainerImage:busybox:1.28 WebServerTestImage:}
[k8s.io] Container runtime should support basic operations on container
  runtime should support listing stats for three created containers when filter is nil. [Conformance]
  github.com/kubernetes-sigs/cri-tools/pkg/validate/container.go:222
[BeforeEach] [k8s.io] Container
  github.com/kubernetes-sigs/cri-tools/pkg/framework/framework.go:50
[BeforeEach] [k8s.io] Container
  github.com/kubernetes-sigs/cri-tools/pkg/validate/container.go:63
[BeforeEach] runtime should support basic operations on container
  github.com/kubernetes-sigs/cri-tools/pkg/validate/container.go:72
[It] runtime should support listing stats for three created containers when filter is nil. [Conformance]
  github.com/kubernetes-sigs/cri-tools/pkg/validate/container.go:222
STEP: create first container
STEP: Get image status for image: busybox:1.28
STEP: Create container.
Jan 19 04:00:02.419: INFO: Created container "0dea6c0df0ce53a9e04a68288a199e95e22c5dda856bb1e48f796c9e0639db60"

STEP: create second container
STEP: Get image status for image: busybox:1.28
STEP: Create container.
Jan 19 04:00:02.986: INFO: Created container "dc8e532aa2846c1007e73203ebbb010b5460d418b2a4049f118d3d4a8398979a"

STEP: create third container
STEP: Get image status for image: busybox:1.28
STEP: Create container.
Jan 19 04:00:03.520: INFO: Created container "ff2c9cdc012af77dacd28de66425a31faf71e035c74470513fca38601a47014c"

STEP: start first container
STEP: Start container for containerID: 0dea6c0df0ce53a9e04a68288a199e95e22c5dda856bb1e48f796c9e0639db60
Jan 19 04:00:03.721: INFO: Started container "0dea6c0df0ce53a9e04a68288a199e95e22c5dda856bb1e48f796c9e0639db60"

STEP: start second container
STEP: Start container for containerID: dc8e532aa2846c1007e73203ebbb010b5460d418b2a4049f118d3d4a8398979a
Jan 19 04:00:03.897: INFO: Started container "dc8e532aa2846c1007e73203ebbb010b5460d418b2a4049f118d3d4a8398979a"

STEP: start third container
STEP: Start container for containerID: ff2c9cdc012af77dacd28de66425a31faf71e035c74470513fca38601a47014c
Jan 19 04:00:04.056: INFO: Started container "ff2c9cdc012af77dacd28de66425a31faf71e035c74470513fca38601a47014c"

STEP: test containers stats
STEP: List container stats for all containers:
[AfterEach] runtime should support basic operations on container
  github.com/kubernetes-sigs/cri-tools/pkg/validate/container.go:76
STEP: stop PodSandbox
STEP: delete PodSandbox
[AfterEach] [k8s.io] Container
  github.com/kubernetes-sigs/cri-tools/pkg/framework/framework.go:51

• [SLOW TEST:5.120 seconds]
[k8s.io] Container
github.com/kubernetes-sigs/cri-tools/pkg/framework/framework.go:72
  runtime should support basic operations on container
  github.com/kubernetes-sigs/cri-tools/pkg/validate/container.go:68
    runtime should support listing stats for three created containers when filter is nil. [Conformance]
    github.com/kubernetes-sigs/cri-tools/pkg/validate/container.go:222
------------------------------
SS
------------------------------
[k8s.io] Container runtime should support basic operations on container
  runtime should support listing stats for started containers [Conformance]
  github.com/kubernetes-sigs/cri-tools/pkg/validate/container.go:195
[BeforeEach] [k8s.io] Container
  github.com/kubernetes-sigs/cri-tools/pkg/framework/framework.go:50
[BeforeEach] [k8s.io] Container
  github.com/kubernetes-sigs/cri-tools/pkg/validate/container.go:63
[BeforeEach] runtime should support basic operations on container
  github.com/kubernetes-sigs/cri-tools/pkg/validate/container.go:72
[It] runtime should support listing stats for started containers [Conformance]
  github.com/kubernetes-sigs/cri-tools/pkg/validate/container.go:195
STEP: create container
STEP: Get image status for image: busybox:1.28
STEP: Create container.
Jan 19 04:00:06.806: INFO: Created container "2eb61d05369373530bbc4b11097b070f5398dcf717cd2e13a852dc6f717d5f36"

STEP: start container
STEP: Start container for containerID: 2eb61d05369373530bbc4b11097b070f5398dcf717cd2e13a852dc6f717d5f36
Jan 19 04:00:06.994: INFO: Started container "2eb61d05369373530bbc4b11097b070f5398dcf717cd2e13a852dc6f717d5f36"

STEP: test container stats
STEP: List container stats for all containers:
[AfterEach] runtime should support basic operations on container
  github.com/kubernetes-sigs/cri-tools/pkg/validate/container.go:76
STEP: stop PodSandbox
STEP: delete PodSandbox
[AfterEach] [k8s.io] Container
  github.com/kubernetes-sigs/cri-tools/pkg/framework/framework.go:51
•SSSSSSSSSS
------------------------------
[k8s.io] Container runtime should support basic operations on container
  runtime should support listing stats for started containers when filter is nil [Conformance]
  github.com/kubernetes-sigs/cri-tools/pkg/validate/container.go:210
[BeforeEach] [k8s.io] Container
  github.com/kubernetes-sigs/cri-tools/pkg/framework/framework.go:50
[BeforeEach] [k8s.io] Container
  github.com/kubernetes-sigs/cri-tools/pkg/validate/container.go:63
[BeforeEach] runtime should support basic operations on container
  github.com/kubernetes-sigs/cri-tools/pkg/validate/container.go:72
[It] runtime should support listing stats for started containers when filter is nil [Conformance]
  github.com/kubernetes-sigs/cri-tools/pkg/validate/container.go:210
STEP: create container
STEP: Get image status for image: busybox:1.28
STEP: Create container.
Jan 19 04:00:09.121: INFO: Created container "3ff79f6b267b676f3074b84e533900c257fa89c37dc0bc61a1842b3c6b0be8b0"

STEP: start container
STEP: Start container for containerID: 3ff79f6b267b676f3074b84e533900c257fa89c37dc0bc61a1842b3c6b0be8b0
Jan 19 04:00:09.272: INFO: Started container "3ff79f6b267b676f3074b84e533900c257fa89c37dc0bc61a1842b3c6b0be8b0"

STEP: test container stats
STEP: List container stats for all containers:
[AfterEach] runtime should support basic operations on container
  github.com/kubernetes-sigs/cri-tools/pkg/validate/container.go:76
STEP: stop PodSandbox
STEP: delete PodSandbox
[AfterEach] [k8s.io] Container
  github.com/kubernetes-sigs/cri-tools/pkg/framework/framework.go:51
•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[k8s.io] Container runtime should support basic operations on container
  runtime should support listing container stats [Conformance]
  github.com/kubernetes-sigs/cri-tools/pkg/validate/container.go:180
[BeforeEach] [k8s.io] Container
  github.com/kubernetes-sigs/cri-tools/pkg/framework/framework.go:50
[BeforeEach] [k8s.io] Container
  github.com/kubernetes-sigs/cri-tools/pkg/validate/container.go:63
[BeforeEach] runtime should support basic operations on container
  github.com/kubernetes-sigs/cri-tools/pkg/validate/container.go:72
[It] runtime should support listing container stats [Conformance]
  github.com/kubernetes-sigs/cri-tools/pkg/validate/container.go:180
STEP: create container
STEP: Get image status for image: busybox:1.28
STEP: Create container.
Jan 19 04:00:11.700: INFO: Created container "0ea95b87caaa82915540726427696219e9dde0796303806c8ca6692585897d10"

STEP: start container
STEP: Start container for containerID: 0ea95b87caaa82915540726427696219e9dde0796303806c8ca6692585897d10
Jan 19 04:00:11.892: INFO: Started container "0ea95b87caaa82915540726427696219e9dde0796303806c8ca6692585897d10"

STEP: test container stats
STEP: List container stats for containerID: 0ea95b87caaa82915540726427696219e9dde0796303806c8ca6692585897d10
[AfterEach] runtime should support basic operations on container
  github.com/kubernetes-sigs/cri-tools/pkg/validate/container.go:76
STEP: stop PodSandbox
STEP: delete PodSandbox
[AfterEach] [k8s.io] Container
  github.com/kubernetes-sigs/cri-tools/pkg/framework/framework.go:51
•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
Ran 4 of 89 Specs in 12.354 seconds
SUCCESS! -- 4 Passed | 0 Failed | 0 Pending | 85 Skipped

You're using deprecated Ginkgo functionality:
=============================================
Ginkgo 2.0 is under active development and will introduce several new features, improvements, and a small handful of breaking changes.
A release candidate for 2.0 is now available and 2.0 should GA in Fall 2021.  Please give the RC a try and send us feedback!
  - To learn more, view the migration guide at https://github.com/onsi/ginkgo/blob/ver2/docs/MIGRATING_TO_V2.md
  - For instructions on using the Release Candidate visit https://github.com/onsi/ginkgo/blob/ver2/docs/MIGRATING_TO_V2.md#using-the-beta
  - To comment, chime in at https://github.com/onsi/ginkgo/issues/711

  You are using a custom reporter.  Support for custom reporters will likely be removed in V2.  Most users were using them to generate junit or teamcity reports and this functionality will be merged into the core reporter.  In addition, Ginkgo 2.0 will support emitting a JSON-formatted report that users can then manipulate to generate custom reports.

  If this change will be impactful to you please leave a comment on https://github.com/onsi/ginkgo/issues/711
  Learn more at: https://github.com/onsi/ginkgo/blob/ver2/docs/MIGRATING_TO_V2.md#removed-custom-reporters
  Measure is deprecated and will be removed in Ginkgo V2.  Please migrate to gomega/gmeasure.
  Learn more at: https://github.com/onsi/ginkgo/blob/ver2/docs/MIGRATING_TO_V2.md#removed-measure
    github.com/kubernetes-sigs/cri-tools/pkg/benchmark/container.go:54
    github.com/kubernetes-sigs/cri-tools/pkg/benchmark/container.go:98
    github.com/kubernetes-sigs/cri-tools/pkg/benchmark/image.go:68
    github.com/kubernetes-sigs/cri-tools/pkg/benchmark/image.go:94
    github.com/kubernetes-sigs/cri-tools/pkg/benchmark/pod.go:42
    github.com/kubernetes-sigs/cri-tools/pkg/benchmark/pod.go:86
    github.com/kubernetes-sigs/cri-tools/pkg/benchmark/pod_container.go:48

To silence deprecations that can be silenced set the following environment variable:
  ACK_GINKGO_DEPRECATIONS=1.16.5

PASS
root@ubuntu-bm:~/go/src/github.com/kubernetes-sigs/cri-tools#

@minbaev
Copy link
Contributor Author

minbaev commented Jan 19, 2022

/retest

@mikebrow
Copy link
Contributor

@minbaev the tests seems to fail, can we fix them?

hey sasha .. chatted with alexander off line.. the fix for this test case still needs the cherry pick over on containerd.. (then move to containerd 1.5.10 when it's out (wip)) or we can run this test against containerd main.. or wait for containerd 1.6.0 which is imminent and has all kinds of k/k 1.23 goodness :-)

@minbaev minbaev force-pushed the add-list-stat-test branch from 126c9db to 151da96 Compare January 21, 2022 16:43
@minbaev minbaev mentioned this pull request Jan 21, 2022
@minbaev minbaev marked this pull request as draft January 21, 2022 18:32
@k8s-ci-robot k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 21, 2022
Copy link
Contributor

@mikebrow mikebrow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

dependency error for the windows build is not related to this PR

we can move to containerd 1.5.10 or 1.6 when they become available, this commit off the top of 1.5 is fair to use for now

@minbaev
Copy link
Contributor Author

minbaev commented Feb 7, 2022

@saschagrunert could you please look at the tests failing. not sure the failure is related to the changes made by this pr

https://github.com/kubernetes-sigs/cri-tools/runs/4900343760?check_suite_focus=true#step:12:112

on windows

 failed to create PodSandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.1": no matching manifest for windows/amd64 10.0.17763 in the manifest list entries

@saschagrunert saschagrunert marked this pull request as ready for review February 8, 2022 08:36
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 8, 2022
@saschagrunert
Copy link
Member

@saschagrunert could you please look at the tests failing. not sure the failure is related to the changes made by this pr

https://github.com/kubernetes-sigs/cri-tools/runs/4900343760?check_suite_focus=true#step:12:112

on windows

 failed to create PodSandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.1": no matching manifest for windows/amd64 10.0.17763 in the manifest list entries

Yes, pause is not available for windows right now. We should exclude those tests for that platform or modify them.

@saschagrunert
Copy link
Member

saschagrunert commented May 2, 2022

@minbaev may I ask you to rebase to get the latest CI changes in?

@minbaev minbaev force-pushed the add-list-stat-test branch from dffdac5 to f6e7658 Compare May 18, 2022 04:59
Copy link
Member

@saschagrunert saschagrunert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@k8s-ci-robot k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels May 23, 2022
@saschagrunert
Copy link
Member

The tests do not seem to work on windows. @minbaev can you take a look?
/lgtm cancel

@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 23, 2022
@minbaev
Copy link
Contributor Author

minbaev commented May 23, 2022

@saschagrunert Hi, what would be the mechanism to exclude the failing windows test due to "k8s.gcr.io/pause:3.1" pulling error? should it be a part of this pr?

@saschagrunert
Copy link
Member

@saschagrunert Hi, what would be the mechanism to exclude the failing windows test due to "k8s.gcr.io/pause:3.1" pulling error? should it be a part of this pr?

Hm, I'm wondering why it works on the master branch compared to this PR 🤔

@mikebrow
Copy link
Contributor

mikebrow commented May 23, 2022

"critest dockershim / windows-2019 (pull_request) Failing after 7m — windows-2019" ..

Looks to me like it's on the wrong pause tag.. 3.1 sounds .. really old probably needs 3.2 - 3.6? dockershim tests were removed...

oh.. looks like the rebase didn't take ..

my rebase workflow

git checkout master
git fetch upstream master --tags
git merge upstream/master
git push origin master
git checkout add-list-stat-test
git rebase master
... fix up any conflicts

@minbaev minbaev force-pushed the add-list-stat-test branch from f6e7658 to 4f533d5 Compare May 23, 2022 16:45
@minbaev
Copy link
Contributor Author

minbaev commented May 23, 2022

@mikebrow @saschagrunert thanks for looking into that
i have followed the flow to rebase with master, but still seeing that pause:3.1 issue. i'm not sure how 3.1 is set
does it maybe have to do with us setting version of containerd to 1.5.9 in the pr whereas it's 1.6.4 already

@mikebrow
Copy link
Contributor

@mikebrow @saschagrunert thanks for looking into that
i have followed the flow to rebase with master, but still seeing that pause:3.1 issue. i'm not sure how 3.1 is set
does it maybe have to do with us setting version of containerd to 1.5.9 in the pr whereas it's 1.6.4 already

check the list of test buckets.. there is no "critest dockershim" test in master, but you have it in your PR's test.. thus I think your PR is not based on the current master...? or I'm reading something wrong...

@minbaev
Copy link
Contributor Author

minbaev commented May 23, 2022

@minbaev
Copy link
Contributor Author

minbaev commented May 23, 2022

also this pr to remove dockershim, but it isn't merged
#871

@minbaev
Copy link
Contributor Author

minbaev commented May 23, 2022

should i just skip the failing tests in dockershim.yml file?

@mikebrow
Copy link
Contributor

should i just skip the failing tests in dockershim.yml file?

nod.. just add your new 3 buckets to the skip.. since most of them are already being skipped

or you could figure out how to move up the pause to 3.something that works.. Guessing 3.1 doesn't have the windows pause in it.. needs to be at least 3.2 would be my guess.

@mikebrow
Copy link
Contributor

also this pr to remove dockershim, but it isn't merged
#871

you know it says dockershim but it's really Cloning into 'D:\a\cri-tools\cri-tools/src/github.com/Mirantis/cri-dockerd'

let's see... ah here it is.. and it's already being fixed Mirantis/cri-dockerd#46

@minbaev minbaev requested a review from saschagrunert May 23, 2022 19:27
Copy link
Contributor

@mikebrow mikebrow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Member

@feiskyer feiskyer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 24, 2022
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: feiskyer, mikebrow, minbaev, saschagrunert

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [feiskyer,saschagrunert]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit 55e46f5 into kubernetes-sigs:master May 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants