Issue #565 bis

[juherr]

Fix #1477

In fact, the issue still exists on travis but not on my computer (the added test is failing).

So, I applied the previous fix (still from #570) and, except the 2 tests I had to modify, it looks to work.

For the record, the 2 tests are really strange:
-> GroupBugTest is working as expected on Travis but not on my computer
-> GuiceParentModuleTest is failing on my computer: I had "parent-module-suite (0)" when "parent-module-suite" is expected, and the opposite 🤦‍♂️

@selenareneephillips Fyi, I have random failures with test.thread.parallelization.ParallelByMethodsTestCase8Scenario1.verifyThatTestMethodsRunInParallelThreads() and test.thread.parallelization.ParallelByMethodsTestCase6Scenario1.verifyThatTestMethodsRunInParallelThreads() if you have any idea.

[juherr]

In fact, it is working well without the fix on teamcity too... https://teamcity.jetbrains.com/viewLog.html?buildId=1117282&tab=buildChangesDiv&buildTypeId=TestNG_BuildTestsWithGradle
So, I can't say if the fix is fixing something or not... 😭

[selenareneephillips]

Hi Julien, Sorry to hear about the random failures. Feel free to disable the tests until I can look into it further. I am in the middle of a big effort on my job that isn't leaving any spare time to do this until the end of the week.

…

-Selena

On Sun, Jul 16, 2017 at 12:28 PM, Julien Herr ***@***.***> wrote: In fact, it is working well without the fix on teamcity too... https://teamcity.jetbrains.com/viewLog.html?buildId= 1117282&tab=buildChangesDiv&buildTypeId=TestNG_BuildTestsWithGradle So, I can't say if the fix is fixing something or not... 😭 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1478 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AD6mAVA3Ly4HSe1MB41LHnK7yQzSvjjVks5sOjoigaJpZM4OZTKq> .

[krmahadevan]

@selenareneephillips - I can give this a try.. But I wasn't able to completely understand the way the parallel tests have been built. Any pointers on from where I should start in the code..? That would help me in understanding the tests and also try and figure out where they are failing.

[selenareneephillips]

@krmahadevan

You should start with the TestNgRunStateTracker class and the TestNgRunStateListener class. That is how I track which methods are run and when. The listener class just logs the events in the order they occur along with test class name, class instance, suite name, test name, method name and thread ID information.

The test methods in the sample classes also all create an event log for the execution of the method body where I also collect information about data provider parameters if the method has a data provider. This is where a lot of bugs in my tests show up -- because I copied and pasted from some other sample class and didn't update the test method name that is logged for the event. Usually I catch this before I check code in.

Other places where dumb mistakes could creep in -- make sure the setup in the test classes matches up with the information I pass to the methods in the base parallelization test class method that verifies that methods are executing in parallel -- thread count, invocation count (if there is a data provider) and so on.

The BaseParallelizationTest is really complex, but this where most of the logic for checking the parallel behavior is implemented. Each test class sets up 1, 2 or 3 suites. The test plan is here:

https://github.com/cbeust/testng/wiki/Parallelization-Test-Plan

It looks like the intermittent failures are coming from these tests:

PTP-TC-4: Parallel by methods mode with parallel test suites using a non-parallel data provider but no dependencies and no factories.

PTP-TC-6: Parallel by methods mode with parallel test suites using factories but no dependencies and no data providers.

PTP-TC-8: Parallel by methods mode with parallel test suites using factories with data providers but no dependencies.

All the failing tests seem to involve the parallel suites which have parallel by methods scenarios.

The method that you need to look into is this one from the BaseParallelizationTest class:

//This deals with tests where there are dataproviders which mean some test methods are run multiple times
public static void verifyParallelTestMethodsWithNonParallelDataProvider(List testMethodEventLogs, String
testName, Map<String, Integer> expectedInvocationCounts, int numUniqueMethods, int
maxSimultaneousTestMethods)

//This deals with tests where there are no dataproviders
public static void verifySimultaneousTestMethods(List testMethodEventLogs, String testName, int
maxSimultaneousTestMethods) {

Anyway -- methods which are run multiple times b/c of data providers are done differently. Each invocation of that method occurs in the same thread and the invocations run sequentially. verifyParallelTestMethodsWithNonParallelDataProvider and verifySimultaneousTestMethods expect there to be a chunk of methods executing at once, based on the thread count setting for the suite and the number of methods remaining. So, in expectation with number of simultaneously executing methods, it takes the event logs in chunks of that size and verifies that the events run in separate threads.

For a test involving no data providers, if I expect there to be 3 methods executing at once, then I expect my current chunk of event logs to show first three method start events, then three method execution events and then three method pass events.

It's more complicated when there are data providers involved because methods are spawned in staggered fashion because there are varying numbers of invocations.

[selenareneephillips]

@krmahadevan

I think one easy improvement to make this easier to diagnose would be statements for pretty printing better failure messages. I kept meaning to go back and make that output more readable, but I didn't get around to it.

[selenareneephillips]

@krmahadevan, @juherr

I am going to implement logging for the parallelization test suite on my commute to and from work so I can diagnose the problem on TeamCity. It looks like java.util.logging is the method of choice for logging in TestNG. Is there some way to view log files generated by test runs on TeamCity? I checked out the latest master branch and I can't reproduce the failures in my test suite locally. These failures have something to do with TeamCity specifically, and I need detailed logging information about the tests in order to track down why they are failing in that environment instead of mine.

[juherr]

@selenareneephillips I think the easier way is using System.out. and you can find the build log like https://teamcity.jetbrains.com/viewLog.html?buildId=1116912&buildTypeId=TestNG_BuildTestsWithGradle&tab=buildLog

But it won't be easy to fix because the last build didn't have any parallel test issue... If you want to try, travis has often random failure and I've never try to understand why.