Add smoke test framework for opensearch bundle #5185
Conversation
Signed-off-by: Zelin Hao <[email protected]>
zelinh
requested review from
dblock,
peterzhuamazon,
gaiksaya,
rishabh6788,
prudhvigodithi,
Divyaasm and
tianleh
as code owners
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #5185 +/- ##
==========================================
- Coverage 92.12% 91.81% -0.31%
==========================================
Files 197 202 +5
Lines 6817 7026 +209
==========================================
+ Hits 6280 6451 +171
- Misses 537 575 +38 ☔ View full report in Codecov by Sentry. |
|
Looking into these check failures. |
Signed-off-by: Zelin Hao <[email protected]>
Signed-off-by: Zelin Hao <[email protected]>
Signed-off-by: Peter Zhu <[email protected]>
Seems like missing bunch of test cases. |
No sure why it's not updating. I attached respective tests after this comments but it still shows 0% |
|
Pending #5187 |
|
Ready for review. |
Thanks for help on fixing the container. |
|
Tested on this docker image |
There was a problem hiding this comment.
Few questions and suggestions:
- How about reusing the current integration test cluster set up instead of installing a cluster for smoke testing? Let me know if the set up is different.
- Would we have a new test-report.yml for smoke testing or will the data be appended to current report.yml?
Adding @dblock to take a look from open-api perspective.
Thanks!
| @@ -0,0 +1,99 @@ | |||
| # Copyright OpenSearch Contributors | |||
There was a problem hiding this comment.
I believe we are duplicating the work of setting up clusters here. Why not just use the integ-test cluster set up instead of creating a new framework for smoke testing? Check the set ups https://github.com/opensearch-project/opensearch-build/tree/main/src/test_workflow/integ_test and see if you can use the same classes to set up a cluster for smoke testing as well. From high level I do not see a difference in the set up.
There was a problem hiding this comment.
Reusing the entire integ-test cluster might be overcomplicate for smoke test as we don't want to accept any customized configuration for smoke tests cluster to keep it lightweight. I also want to make smoke test workflow less dependent on integ tests as a standalone test workflow.
There was a problem hiding this comment.
For APIs to work at component level we might need additional-config to be added.
For example: You cannot check system templates plugin if these configs are not enabled.
The current set up for integ-test is light weight as well. Reusing the existing codebase might be the way to go instead of reinventing the wheel for different usecase. Also per dustribution it will go on increasing like the way we have today for integ-test.
There was a problem hiding this comment.
Any example API that I can test for system templates plugin you are referring? Anyway I think if certain component requires specific configuration for smoke test, it's not ready to be onboarded to smoke tests framework. Because we only deploy the cluster once and run all API request checks against it. We are not sure whether that specific configuration would affect others.
We are reusing the distributions classes from integ tests workflow to install and start the cluster. For any future distribution type, it can be easily adopted by this smoke test workflow.
There was a problem hiding this comment.
Agree with reusing the cluster setup from integ test, the code is pretty modular to be reused here. To be able to even run an api of system-templates you need those configurations added to opensearch.yml.
Think about adding smoke tests for features that need to be enabled explicitly.
There was a problem hiding this comment.
I still think we shouldn't allow any customized configurations as it might grow much bigger. We provide generic framework with all default configurations for the cluster by design. If any of the component needs specific configuration to be operational, it shouldn't be added to the smoke tests and can go into the integ tests.
There was a problem hiding this comment.
We can start with this approach but overall I think we will need the additional config for plugins specifically. Another use-case is if a new plugin or feature is added that is experimental, we might need to enable it using additional-config.
Again this is conditional to defining what qualifies as smoke testing.
There was a problem hiding this comment.
The requirements of smoke tests is set for default configuration and we don't want any additional configuration affect others test cases. Since smoke tests won't be the only exit criteria for RC, i think these tests cases can and shall be added to integ tests with more configuration options. Anyway, we may discuss more on that as an enhancement.
| @@ -0,0 +1,30 @@ | |||
| # yamllint disable | |||
There was a problem hiding this comment.
How about storing these files under manifests folder per version?
manifests/2.18.0/smoke_test/opensearch.yml
There was a problem hiding this comment.
At this point, i don't feel it's necessary to keep these files distinct by version. So far the API requests in this file are generic and not version specific. In addition, any of the valid API path can be filtered in the API-spec file for the first version it was introduced.
There was a problem hiding this comment.
For new APIs that are being added how do you propose to manage that?
There was a problem hiding this comment.
The version introduced will be specified in the API-spec file and we can check the version and compare it with the current version.
There was a problem hiding this comment.
Sorry for not being clear, what I meant was if we introduce new APIs in say 2.19.0 would we be using same spec file that we used in 2.18.0?
Wondering how api-spec repo is managing that today.
There was a problem hiding this comment.
This component spec will still be the same. We check on the test workflow side.
The workflow extracts the api path from this component spec, checks current cluster version vs the x-version-add from api-spec and run checks if current version is later than added version.
There was a problem hiding this comment.
if not differentiated on version it will become a problem when 3.0 is being prepared to release and has breaking changes in existing apis.
You don't have to separate at each minor but atleast at 2.x and 3.x. Minor versions are backward compatible so it is safe to assume there will be no breaking changes in the apis.
We don't have to worry about introducing new apis in minor versions as smoke tests are expected to run for versions to be released and not already released versions.
There was a problem hiding this comment.
I'm okay if openapi handles the version check and returns appropriate response.
Say, version 3.0 made a change to _cat/indices api request/response and you run this workflow for both 2.x and 3.x for a new release, how will that be handled?
There was a problem hiding this comment.
Those changes should also be handled in the openapi spec files.
There was a problem hiding this comment.
@gaiksaya @rishabh6788 I updated these component spec by versions to accommodate any major version breaking API changes. Please help review it again. Thanks!
|
| url = "https://localhost:9200/" | ||
| logging.info(f"Pinging {url}") | ||
| try: | ||
| request = requests.get(url, verify=False, auth=("admin", "myStrongPassword123!")) |
There was a problem hiding this comment.
Would it always be with-security?
There was a problem hiding this comment.
Would recommend to create an issue for without-security as security is an optional plugin and smoke tests just like integ-test need to run smoothly irrespective of security plugin
Signed-off-by: Zelin Hao <[email protected]>
| - name: opensearch | ||
| smoke-test: | ||
| test-spec: opensearch.yml |
There was a problem hiding this comment.
We haven't started the release cycle for 2.19 so there is no complete distribution bundle for testing purpose yet so I used 2.18 to demonstrate the correct command to start the workflow. I'm OK to change this to 2.19; and we should also update the manifest workflow to attach the smoke tests entry into the test manifest creation.
There was a problem hiding this comment.
Moved to 2.19 test manifest.
There was a problem hiding this comment.
Confirmed new command ./test.sh smoke-test manifests/2.19.0/opensearch-2.19.0-test.yml --paths opensearch=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.19.0/10545/linux/x64/tar/ is running correctly.
Signed-off-by: Zelin Hao <[email protected]>
Signed-off-by: Zelin Hao <[email protected]>
Signed-off-by: Zelin Hao <[email protected]>
| # Only copy if it's a file | ||
| if os.path.isfile(src_path): | ||
| shutil.copy2(src_path, dest_path) | ||
| print(f"Copied {src_path} to {dest_path}") |
There was a problem hiding this comment.
Replace with logging instead of printing
| super().__init__(args, test_manifest) | ||
| logging.info("Entering Smoke test for OpenSearch Bundle.") | ||
|
|
||
| # TODO: Download the spec from https://github.com/opensearch-project/opensearch-api-specification/releases/download/main-latest/opensearch-openapi.yaml |
There was a problem hiding this comment.
Maybe create an issue with api-spec repo to publish releases frequently?
Signed-off-by: Zelin Hao <[email protected]>
Description
Add smoke test framework for opensearch bundle
The smoke test workflow can be started with both local bundle artifacts or from CI.
Example command:
./test.sh smoke-test manifests/2.18.0/opensearch-2.18.0-test.yml --paths opensearch=/test/taror
./test.sh smoke-test manifests/2.18.0/opensearch-2.18.0-test.yml --paths opensearch=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.18.0/10479/linux/x64/tar/The essential functionalities of this smoke test workflow is to deploy a staging bundle cluster by using
distributionclass form integ test workflow, run some basic API requests and validate its response schema against theopensearch-api-specification.Issues Resolved
#5164
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.