TestNG reports not getting generated in Gradle

[itkhanz]

Hi,
I am facing this strange issue where if the build duration is longer (1 hour+) and have many tests (50+) then Gradle does not generate TestNG reports. Intrestingly, this issue does not happen if the build duration is small and have only fewer tests.

I wonder if it has something to do with the way TestNG report generation works with Gradle build tool? I am not sure whether this issue is happening because of TestNG or Gradle.

Here is a link to my post on Gradle discussion form where you can find more details:
https://discuss.gradle.org/t/gradle-does-not-generate-testng-report-for-longer-builds/48433

Could you please take a look and suggest what could possibly be causing this issue? and suggest the right approach to make sure that TestNG reports will always be generated after gradle build finishes.

Thank you.

[krmahadevan]

@itkhanz - Have you looked at this line here

2024-05-09T20:10:35.740+0200 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] > There were failing tests. See the report at: file:///Users/ibtisam.khan/dcs/dev/dcs-app-automated-tests/build/reports/tests/test/index.html

Not sure what we can do from TestNG side here unless and until you can simulate the problem using a command line invocation

[itkhanz]

Yes but i think this hints to failing tests in gradle task, so I assume this is not the issue as this line simply means that some tests got failed and links to Gradle generated index.html
This debug statement also gets printed even when there is a successful TestNG report created.

[krmahadevan]

@itkhanz - Could be. Hard to say. I would suggest that you first try to simulate this problem with just TestNG. What happens if you create a custom Gradle task that explicitly invokes TestNG?

Something like (My Gradle skills aren't that great. So feel free to tweak this to match the actual syntax)

task runTestNGTests() {
    doLast {
        def testng = new org.testng.TestNG()
        testng.setOutputDirectory('build/test-results/testng')
        testng.setTestClassesDir(sourceSets.test.output.classesDirs.singleFile)
        testng.setVerbose(3)
        testng.run()
    }
}

[itkhanz]

Thank you for looking into this issue. I was able to find out reason why TestNG reports were not being produced. It turned out, for some of my tests, NullPointerException was being thrown in the test listener causing Gradle to throw this below exception and not generating TestNG reports
org.gradle.api.internal.tasks.testing.TestSuiteExecutionException: Could not complete execution for Gradle Test Executor 30.
It seems to me a more gradle specific issue where it was not able to handle this exception coming from testng listener class implementation.

On my end, I was able to resolve this issue by implementing a nullsafe mechanism so tests now don't throw NullPointerException leading to successfull creation of testng-results.xml report and other TestNG reports.

I have mentioned the steps to reproduce and details in the issue I created at gradle:
gradle/gradle#29098 (comment)