Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extend FAIL_ON_SKIP To gtest #640

Open
lplewa opened this issue Jul 26, 2024 · 0 comments
Open

Extend FAIL_ON_SKIP To gtest #640

lplewa opened this issue Jul 26, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@lplewa
Copy link
Contributor

lplewa commented Jul 26, 2024

Problem Statement

The current CMake option UMF_TESTS_FAIL_ON_SKIP is designed to handle skipped tests. However, it does not work with GTEST_SKIP in Google Test. This limits its functionality and does not enforce consistent behavior for skipped tests.

Objective

Extend the UMF_TESTS_FAIL_ON_SKIP option to work with GTEST_SKIP to ensure that skipped tests are treated as failures when this option is enabled.

Solution

Implement a macro to replace GTEST_SKIP with a custom macro (UMF_SKIP). This macro will conditionally either call GTEST_SKIP or fail the test based on the UMF_FAIL_ON_SKIP option.

Implementation Steps

  1. Define the Custom Macro:

Create a new macro UMF_SKIP which will conditionally handle skipped tests based on the UMF_FAIL_ON_SKIP option.

#ifdef UMF_FAIL_ON_SKIP
#define UMF_SKIP GTEST_FAIL() << "UMF_TESTS_FAIL_ON_SKIP Enabled: Treating skip as fail"
#else
#define UMF_SKIP GTEST_SKIP()
#endif
  1. Replace GTEST_SKIP in Tests:

Replace all occurrences of GTEST_SKIP in tests with UMF_SKIP.

  1. Define UMF_FAIL_ON_SKIP Through Compiler Argument:

Ensure the UMF_FAIL_ON_SKIP macro can be defined via a compiler argument (e.g., -DUMF_FAIL_ON_SKIP).
Modify the CMake configuration to include this option.

if(UMF_TESTS_FAIL_ON_SKIP)
    add_definitions(-DUMF_FAIL_ON_SKIP)
endif()
  1. Update CI Configuration:

Explicitly exclude tests that are expected to skip on the CI environment in the YAML configuration file for CI pipeline.
This is the hardest part, as we do not have option to disable single "gtest" test from YAML level. We did something like this for valgrind tests - but it required separate .sh file for those tests, so better approach is to add extra cmake configuration flags <TEST_NAME>_FILTER and modify add_test function to handle it

@lplewa lplewa added the enhancement New feature or request label Jul 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant