Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

configfailurepolicy=continue only works for BeforeTest when using TestNG XML file #2731

Open
1 of 7 tasks
Mei-self opened this issue Feb 17, 2022 · 6 comments · May be fixed by #2735
Open
1 of 7 tasks

configfailurepolicy=continue only works for BeforeTest when using TestNG XML file #2731

Mei-self opened this issue Feb 17, 2022 · 6 comments · May be fixed by #2735

Comments

@Mei-self
Copy link

Mei-self commented Feb 17, 2022

TestNG Version

7.5

Expected behavior

if configfailurepolicy=continue and any BeforeXXX method fails, the remaining test method should execute not skip

Actual behavior

only BeforeTest fails, the remaining test method would execute

Is the issue reproducible on runner?

  • Shell
  • Maven
  • Gradle
  • Ant
  • Eclipse
  • IntelliJ
  • NetBeans

Test case sample

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="TestSuite" configfailurepolicy="continue">
    <test name="Test">
        <classes>
            <class name="TestConfigFail"/>
        </classes>
    </test>
</suite>
public class TestConfigFail {

    @BeforeSuite
    public void beforeSuite() {
        Assert.fail("This before suite is fail");
    }

    @BeforeTest
    public void beforeTest() {
        Assert.fail("This before test is fail");
    }

    @BeforeClass
    public void beforeClass(){
        Assert.fail("This before class is fail");
    }

    @BeforeMethod
    public void beforeMethod(){
        Assert.fail("This before method is fail");
    }

    @Test
    public void test() {
        System.out.println("This is test method");
    }
}

Contribution guidelines

Incase you plan to raise a pull request to fix this issue, please make sure you refer our Contributing section for detailed set of steps.

@krmahadevan
Copy link
Member

@Wang-Meimei - Is this something that was broken in 7.5 or has this never worked ? Can u please check in 7.4.0 and 6.14.3 and post back your findings ?

@Mei-self
Copy link
Author

@Wang-Meimei - Is this something that was broken in 7.5 or has this never worked ? Can u please check in 7.4.0 and 6.14.3 and post back your findings ?

It seems this never worked. But from the document, it should work. So maybe need to update document?

@juherr
Copy link
Member

juherr commented Feb 18, 2022

It seems this never worked. But from the document, it should work. So maybe need to update document?

Yes, as we know now it is no a regression, we need to determine the expected behavior and fix the code or the doc.

I think the issue is valid and the method should be run.
@krmahadevan wdyt?

@krmahadevan
Copy link
Member

It seems this never worked. But from the document, it should work. So maybe need to update document?

Yes, as we know now it is no a regression, we need to determine the expected behavior and fix the code or the doc.

I think the issue is valid and the method should be run. @krmahadevan wdyt?

I dont know if this is an issue @juherr .

If there is a higher and lower order configuration, then we should not try to execute the lower order configuration.

For .e.g,

Lets say I have

  • @BeforeSuite
  • @BeforeTest
  • @Test

In this case, we should not try to execute the @BeforeTest because this is a configuration method. But then one can argue that if one has to get to the level of running a @Test method, then all the above level configurations need to be run.

So am not sure what the behaviour should be.

@bj-9527 - Please note.

@juherr
Copy link
Member

juherr commented Feb 28, 2022

IMO, the only difference between continue and skip is that continue will execute the method, and skip won't execute and will directly set the SKIP status.

But when I see #2735 changes in tests, I wonder if the behavior change should not be configurable.

@bj-9527
Copy link
Contributor

bj-9527 commented Mar 1, 2022

It could be config by FailurePolicy which could be set as SKIP or CONTINUE, then the behavior could change.

So am not sure what the behaviour should be.

For, this change, the SKIP are working as before, the CONTINUE will execute the following method even previous config failed.

bj-9527 added a commit to bj-9527/testng that referenced this issue Mar 1, 2022
bj-9527 added a commit to bj-9527/testng that referenced this issue Mar 1, 2022
bj-9527 added a commit to bj-9527/testng that referenced this issue Mar 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants