Failure when the job contains a lot of tests.

[gvas]

When a job contains a lot of tests then grunt-saucelabs incorrectly reports that the tests have failed. I think this is related to the 64KB size limit put on the custom-data field as mentioned in the SauceLabs documentation in the Record Custom Data section.

I have 72 jasmine tests. The REST API in my case returns the following JSON:

{
    "completed": true,
    "js tests": [
        {
            "url": "http://saucelabs.com/jobs/af0c8d88457d407095ccc824696b5fcf",
            "platform": [
                "Windows XP",
                "chrome",
                ""
            ],
            "result": null,
            "id": "1c12d9ed1967473a9080bfec6a99ff46",
            "job_id": "af0c8d88457d407095ccc824696b5fcf"
        }
    ]
}

Note that the result element's value is null and there is no passed element. This makes grunt-saucelabs to believe that the tests have failed.

Removing some tests fixes the problem (as a symptomatic treatment).

[Jonahss]

All you have to do is change the reporting code on your test page. Change the output of the Jasmine Reporter (jasmine.getJSReport()) to only include failed tests, rather than all tests.

Another option: Break your test suite into two pages. Half the results per page.

This is an issue with the Sauce Labs Unit Test API, not grunt-saucelabs, so I'm closing the issue.

[gvas]

Yes, thanks, the 2nd one was the workaround I have come up with myself, too.
However I think this limitation deserve a line in the README.md and/or a log entry in the grunt task. It required a considerable amount of time to figure out why were my tests failing.

[Jonahss]

How about I add better error reporting on the job, as part of #102

[gvas]

Thank you for expending time on this issue!
What will your fix for #102 will cover? Some general error message like "Unexpected response format" or "There is no 'passed' element in the response"? If so then I expect that there will be new issues asking what its meaning is. But at least there will be something that can be searched for.

[Jonahss]

It will be a combination of better job error reporting, and invalid API
parameters.
So in this case it would be: "max-duration" value invalid, expected int
between 0 and XXX inclusive.

Better job error reporting will be like: "job timed out after XX
milliseconds" or "browser crashed".

grunt-sacuelabs will then pass these forward to the cli.

On Wed, May 21, 2014 at 1:44 PM, gvas [email protected] wrote:

Thank you for expending time on this issue!
What will your fix for #102https://github.com/axemclion/grunt-saucelabs/issues/102will cover? Some general error message like "Unexpected response format" or
"There is no 'passed' element in the response"? If so then I expect that
there will be new issues asking what its meaning is. But at least there
will be something that can be searched for.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/123#issuecomment-43811455
.

[gvas]

I've got another idea. The result element's null value indicates that the 64KB limit is exceeded. When this is detected, given the job id we can download the job's assets including the sauce log, among others. It contains the full test result. I'm going to work on this.

[Jonahss]

Ooh nice. Feel free to submit a PR for that.

I'm hoping to devote time to fixing that problem with Sauce. I'm thinking of truncating the long results. issuing a warning, but the job will still pass/fail correctly.