Gather data about private API usage in test suite #1800
Merged
Commits on Jul 24, 2024
-
Remove shortcuts to shared_helper properties
This refactor is preparation for an upcoming change which will give each test function a unique helper object.
-
Declare variables consistently in shared_helper
Preparation for an upcoming change which will turn this helper into a class.
-
Stop using helper.AblyRealtime as a constructor
It isn’t intended be called as a constructor.
-
Convert shared helper to a class
Motivation as in 9e28603. To expand a little, in order to implement ECO-4821 (gathering detailed information about private API usage in the test suite), we will give each test function access to a unique helper object which is aware of the identity of the test case in which it is executing. This helper object will offer a method for recording the fact that a test function is calling a private API. By using the existing shared_helper object, we can also record the usage of private APIs by the various helper methods, access to which is currently funnelled through the shared_helper. The couple of methods that I’ve made static (randomString, whenPromiseSettles) are ones that are very frequently called _and_ which definitely don’t use any private APIs, so I’ve made them static so that, when we make the aforementioned change giving each test case its own helper object, we don’t need to pass a helper around just to enable these methods to be called. The choice of making just these two methods static was pretty arbitrary; once we’ve catalogued all of the private API usage it might turn out there some other ones that can be made static.
-
Update tests so that each has its own helper instance
This removes the per-file SharedHelper instance and instead makes sure that each context which will call SharedHelper instance methods (e.g. test cases, test hooks) has a unique SharedHelper instance that is aware of the context from which it is being called. (Motivation for this change already explained in 1fa5e13.)
-
Move availableTransports and bestTransport inside helper
Preparation for recording private API usage by tests (ECO-4821).
-
Record usages of private APIs in the test suite
This adds annotations to each place in the test suite where a private API is called. When the tests are run, these annotations are used to emit a JSON file containing private API usage information. (In ECO-4834, we’ll process this JSON file to generate some useful reports.) This JSON file also contains a list of all of the tests which get run, which will be used as part of the aforementioned reporting to allow us to generate a report of the tests that do not use any private APIs. The motivation for this gathering this data is that we intend to try and reuse the ably-js test suite as a unified test suite for all of our client libraries. In order to do this, we’ll need to address the usage of ably-js private APIs in the test suite, and to do that we need to first gather data about the current usage. I originally intended to put the changes into the current commit into a separate unified test suite branch, instead of merging them into `main`. But I don’t really want to maintain a long-running feature branch, and also I want to make sure that we add private API annotations to any new tests that we add to ably-js, hence I decided to put this commit into `main`. But maybe, once we’re done with the unified test suite, we’ll decide to remove these annotations. Given the context in which we’re gathering this data, in the interests of saving time I skipped adding some annotations to some files that I’m pretty sure will not form part of the unified test suite: - test/realtime/transports.test.js - test/browser/simple.test.js - test/browser/http.test.js - test/browser/connection.test.js - test/browser/modular.test.js - test/browser/push.test.js - test/rest/bufferutils.test.js If we decide at some point that we’d like to for whatever reason gather data on these tests, we can do that later. (Note I haven't been very consistent with how many times per test I record the usage of a given private API; it’s not a stat I’m particularly interested in here. But for the most part I’m doing it at the point of each invocation of the private API.) Resolves ECO-4821.
-
Generate reports on private API usage
This uses the data gathered in 2dcf4ed to generate some CSV reports about private API usage in the test suite. We’ll use these reports as a starting point for deciding how to remove this private API usage when reusing the test suite as a unified test suite for all our client libraries. Resolves ECO-4834.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.