-
-
Notifications
You must be signed in to change notification settings - Fork 22
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Avoid random failures in system tests #5319
Draft
jorg-vr
wants to merge
24
commits into
main
Choose a base branch
from
fix/randomly-failing-tests
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This reverts commit cdc755a.
This reverts commit 6609925.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request is an effort to avoid occasional failures in the system tests. The global solution was increasing the maximal wait time for finding elements on a page to 30 seconds.
This does not slow down succeeding test. It will slowdown failing tests.
There were multiple points of failure encountered while working on this pr:
This finally led me to conclude that the issues were not test specific.
This pr also has multiple minor changes to help avoid these random failures:
annotation-text
count to change before checking backend change. This should avoid a timing issues if the frontend changes has not yet propagated to the backend.The tests now use a json exercise with the normal feedback table renderer. The assumption here was that the code tab rendering might have infrequent errors for the code tab.The code tab is clicked reference by id instead of by text. The idea was that a text search for the button may sometimes fails due to the number tag also in the buttonThe maximum waiting time for finding the code tab has been increased to 5 seconds (maybe the page is slow to load)I have added a sanity check to first find the body element and the page wrapper before searching for the code tab. I added this to debug, not as a fix, but it is the only thing that worked as a fix.Increase wait time for scratchpad test. This test deserves some additional worst case wait time for the setup of webworker.I have tested this pr by rerunning the tests 10 times after each change.
Notes
I found it very odd, that although it varies in which test a failure happens, there is always only a single failing test.
Increasing the number of tests, by duplicating one of the failing tests 40 times, did not seem to increase the frequency of failures. (It took me 10 tries to reproduce a failure with the 40 added tests)
When I started working on this issue I mostly encountered the 'code tab not found' failure.
When that error stopped occurring I started seeing the 'question count value incorrect' failure.
Now that that that seems fixed, the #4562 failure reappeared after not having seen it for weeks.
Increasing the allowed waiting time for that test, let another totally unrelated test fail...
Except for the incorrect count the issue is always the same. An item that is present on the page is not found.
Always only a single test failing