Gather data about private API usage in test suite #1800
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
temporarily deployed
to
staging/pull/1800/bundle-report
lawrence-forooghian
force-pushed
the
ECO-4834-do-something-with-private-api-data
branch
from
cb2f0ed
to
dbda180
Compare
github-actions
bot
temporarily deployed
to
staging/pull/1800/bundle-report
lawrence-forooghian
force-pushed
the
ECO-4834-do-something-with-private-api-data
branch
from
dbda180
to
a575337
Compare
github-actions
bot
temporarily deployed
to
staging/pull/1800/bundle-report
lawrence-forooghian
force-pushed
the
ECO-4834-do-something-with-private-api-data
branch
from
a575337
to
4ad5132
Compare
github-actions
bot
temporarily deployed
to
staging/pull/1800/bundle-report
github-actions
bot
temporarily deployed
to
staging/pull/1800/bundle-report
github-actions
bot
temporarily deployed
to
staging/pull/1800/bundle-report
lawrence-forooghian
force-pushed
the
ECO-4834-do-something-with-private-api-data
branch
from
84a4e15
to
1c0126c
Compare
github-actions
bot
temporarily deployed
to
staging/pull/1800/bundle-report
github-actions
bot
temporarily deployed
to
staging/pull/1800/bundle-report
lawrence-forooghian
force-pushed
the
ECO-4834-do-something-with-private-api-data
branch
from
055a8d7
to
9525f54
Compare
lawrence-forooghian
force-pushed
the
ECO-4821-gather-further-private-API-usage-info
branch
from
9986dc4
to
d551ba5
Compare
lawrence-forooghian
force-pushed
the
ECO-4834-do-something-with-private-api-data
branch
from
efc827f
to
266b06e
Compare
github-actions
bot
temporarily deployed
to
staging/pull/1800/bundle-report
lawrence-forooghian
force-pushed
the
ECO-4834-do-something-with-private-api-data
branch
from
266b06e
to
5dd49bc
Compare
github-actions
bot
temporarily deployed
to
staging/pull/1800/bundle-report
lawrence-forooghian
force-pushed
the
ECO-4834-do-something-with-private-api-data
branch
from
5dd49bc
to
58307a7
Compare
github-actions
bot
temporarily deployed
to
staging/pull/1800/bundle-report
This refactor is preparation for an upcoming change which will give each test function a unique helper object.
Preparation for an upcoming change which will turn this helper into a class.
It isn’t intended be called as a constructor.
Motivation as in 369ad7e. To expand a little, in order to implement ECO-4821 (gathering detailed information about private API usage in the test suite), we will give each test function access to a unique helper object which is aware of the identity of the test case in which it is executing. This helper object will offer a method for recording the fact that a test function is calling a private API. By using the existing shared_helper object, we can also record the usage of private APIs by the various helper methods, access to which is currently funnelled through the shared_helper. The couple of methods that I’ve made static (randomString, whenPromiseSettles) are ones that are very frequently called _and_ which definitely don’t use any private APIs, so I’ve made them static so that, when we make the aforementioned change giving each test case its own helper object, we don’t need to pass a helper around just to enable these methods to be called. The choice of making just these two methods static was pretty arbitrary; once we’ve catalogued all of the private API usage it might turn out there some other ones that can be made static.
This removes the per-file SharedHelper instance and instead makes sure that each context which will call SharedHelper instance methods (e.g. test cases, test hooks) has a unique SharedHelper instance that is aware of the context from which it is being called. (Motivation for this change already explained in 18613c3.)
Preparation for recording private API usage by tests (ECO-4821).
This adds annotations to each place in the test suite where a private API is called. When the tests are run, these annotations are used to emit a JSON file containing private API usage information. (In ECO-4834, we’ll process this JSON file to generate some useful reports.) This JSON file also contains a list of all of the tests which get run, which will be used as part of the aforementioned reporting to allow us to generate a report of the tests that do not use any private APIs. The motivation for this gathering this data is that we intend to try and reuse the ably-js test suite as a unified test suite for all of our client libraries. In order to do this, we’ll need to address the usage of ably-js private APIs in the test suite, and to do that we need to first gather data about the current usage. I originally intended to put the changes into the current commit into a separate unified test suite branch, instead of merging them into `main`. But I don’t really want to maintain a long-running feature branch, and also I want to make sure that we add private API annotations to any new tests that we add to ably-js, hence I decided to put this commit into `main`. But maybe, once we’re done with the unified test suite, we’ll decide to remove these annotations. Given the context in which we’re gathering this data, in the interests of saving time I skipped adding some annotations to some files that I’m pretty sure will not form part of the unified test suite: - test/realtime/transports.test.js - test/browser/simple.test.js - test/browser/http.test.js - test/browser/connection.test.js - test/browser/modular.test.js - test/browser/push.test.js - test/rest/bufferutils.test.js If we decide at some point that we’d like to for whatever reason gather data on these tests, we can do that later. (Note I haven't been very consistent with how many times per test I record the usage of a given private API; it’s not a stat I’m particularly interested in here. But for the most part I’m doing it at the point of each invocation of the private API.) Resolves ECO-4821.
This uses the data gathered in 3a86ca3 to generate some CSV reports about private API usage in the test suite. We’ll use these reports as a starting point for deciding how to remove this private API usage when reusing the test suite as a unified test suite for all our client libraries. Resolves ECO-4834.
lawrence-forooghian
force-pushed
the
ECO-4834-do-something-with-private-api-data
branch
from
58307a7
to
9b90cd5
Compare
lawrence-forooghian
changed the title
VeskeR
requested changes
Jul 19, 2024
There was a problem hiding this comment.
This is an incredible work on private API usage! Couple of small comments below.
I really like the idea of test contexts and the design of PrivateApiRecorder class. I can see us expanding on this idea and adding something like SpecItemCoverageRecorder for aligning tests with the spec from #1806. This will give us better a better overview of which tests are covering specific spec item and how many times (given that we have things like testOnAllTransports we can repeat the test for the same spec item multiple times). And also we would be able to automate checking that each test case has an associated spec item (or tagged as nospec) by checking if it called a corresponding method from SpecItemCoverageRecorder.
Really looking forward to what we can do with this in the future
| }) | ||
| .catch((err) => { | ||
| callback(err); | ||
| async closeAndFinishAsync(realtime, err) { |
There was a problem hiding this comment.
shouldn't this one be _closeAndFinishAsync ?
|
|
||
| function createTest(options) { | ||
| return function (done) { | ||
| this.test.helper = this.test.helper.withParameterisedTestTitle(name); |
There was a problem hiding this comment.
wouldn't we want to use a test name with transport here, like name + '_with_' + transport + '_binary_transport' below in itFn calls? so passing that concatenated string to createTest function
| 'listen.connectionManager.transport.active', | ||
| 'listen.connectionManager.transport.pending', | ||
| 'listen.transport.disposed', | ||
| 'new.Crypto.CipherParams', // This is in a bit of a grey area re whether it’s private API. The CipherParams class is indeed part of the public API in the spec, but it’s not clear whether you’re meant to be construct one directly, and furthermore the spec doesn’t describe it as being exposed as a property on Crypto. |
There was a problem hiding this comment.
Suggested change
| 'new.Crypto.CipherParams', // This is in a bit of a grey area re whether it’s private API. The CipherParams class is indeed part of the public API in the spec, but it’s not clear whether you’re meant to be construct one directly, and furthermore the spec doesn’t describe it as being exposed as a property on Crypto. | |
| 'new.Crypto.CipherParams', // This is in a bit of a grey area re whether it’s private API. The CipherParams class is indeed part of the public API in the spec, but it’s not clear whether you’re meant to construct one directly, and furthermore the spec doesn’t describe it as being exposed as a property on Crypto. |
| keyForValue: (value: Value) => Key, | ||
| areKeysEqual: (key1: Key, key2: Key) => boolean, | ||
| ) { | ||
| const result: Group<Key, Value>[] = []; |
There was a problem hiding this comment.
could we use Map<Key, Value[]> here to simplify the types a bit?
| { key: 'file', value: context.file ?? 'null' }, | ||
| { key: 'suite', value: context.suite?.join(',') ?? 'null' }, | ||
| { key: 'title', value: context.type === 'definition' ? context.label : context.title }, | ||
| { key: 'helperStack', value: 'helperStack' in context ? context.helperStack.join(',') : 'null' }, |
There was a problem hiding this comment.
I think we should sort helperStack here before joining, since I believe we don't want the order in which we called addingHelperFunction to matter here.
| } | ||
| } | ||
|
|
||
| // otherwise, return true iff it’s the same test |
There was a problem hiding this comment.
Suggested change
| // otherwise, return true iff it’s the same test | |
| // otherwise, return true if it’s the same test |
There was a problem hiding this comment.
I was using "iff" as an abreviation for "if and only if" but I think it'd be clearer to write it out in full, thanks!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR collects data about the usage of private APIs in the test suite, and uses this data to generate some reports. We’ll use these reports to guide our work of converting the ably-js test suite into a unified test suite for all of our client libraries. See commit messages for more details.
Resolves ECO-4821, resolves ECO-4834.