From ea23a41db2b8e53e1167b3580388b05716a7b1cd Mon Sep 17 00:00:00 2001 From: Dave Lockhart Date: Wed, 9 Aug 2023 13:56:50 -0400 Subject: [PATCH] running tests documentation --- README.md | 204 ++++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 177 insertions(+), 27 deletions(-) diff --git a/README.md b/README.md index 97a1c469..a9f6e09d 100644 --- a/README.md +++ b/README.md @@ -4,8 +4,6 @@ Testing utilities which are specifically designed and configured for Brightspace UI components and applications. -## Installation - Install from NPM: ```shell @@ -99,7 +97,7 @@ it('should work in RTL', async() => { If `lang` is set to Arabic (`ar`), the right-to-left option will automatically be enabled. -> **Note:** it's not recommended to use `language` configuration with visual-diff to solely test the correctness of translations. The [messageformat-validator](https://github.com/bearfriend/messageformat-validator) is a more efficient way to test translations. +> **Note:** it's not recommended to use `language` configuration with [vdiff](#vdiff-testing) to solely test the correctness of translations. The [messageformat-validator](https://github.com/bearfriend/messageformat-validator) is a more efficient way to test translations. ### Accessibility Testing with aXe @@ -208,7 +206,7 @@ it('should wait for an event', async() => { Note that the call to `clickElem` is not `await`-ed, since by the time it resolves the event will have already been dispatched. -If you need to prevent the default behaviour of the event in your test, you can use `oneDefaultPreventedEvent`. +If the test needs to prevent the default behavior of the event, use `oneDefaultPreventedEvent`. #### Waiting for a Lit Element to Update @@ -239,7 +237,7 @@ it('should wait', async() => { #### Waiting For Asynchronous Components -`fixture()` will automatically wait for all nested Lit components to fulfill their `updateComplete` Promise. To tweak when a component's `updateComplete` fulfills, implement the [`getUpdateComplete()` lifecycle callback](https://lit.dev/docs/components/lifecycle/#getUpdateComplete). +`fixture()` will automatically wait for all nested Lit components to fulfill their `updateComplete` Promise. To control when a component's `updateComplete` fulfills, implement the [`getUpdateComplete()` lifecycle callback](https://lit.dev/docs/components/lifecycle/#getUpdateComplete). In other scenarios, a component may have an initial loading state (e.g. loading spinner or skeleton) where `updateComplete` has already resolved in addition to another fully loaded state. To signal that `fixture()` should wait for this final state, implement `getLoadingComplete()`. It works the same way as `getUpdateComplete()` by fulfilling its Promise when the component has fully loaded. @@ -292,7 +290,7 @@ await waitUntil(() => elem.condition, { ### Defining a Custom Element for Testing -If a test requires a one-off custom element, define it using `defineCE` and pass the returned tag name in the call to `fixture()`. +If a test requires a one-off custom element, define it using `defineCE` and pass the returned tag name to `fixture()`. ```javascript import { defineCE, fixture, html } from '@brightspace-ui/testing'; @@ -316,11 +314,97 @@ it('should use custom element', async() => { > **Important:** `defineCE` is not performant and shouldn't be used outside of test files. -### Vdiff Testing +## Running Tests + +Use the `d2l-test-runner` binary to execute a set of tests and report their results. It builds upon the robust [@web/test-runner](https://modern-web.dev/docs/test-runner/overview/), while configuring it for Brightspace components and applications. + +### CLI and Configuration + +`d2l-test-runner` can be configured using CLI flags or an optional configuration file. -Short for "visual diff" and also known as "visual regression" or "perceptual diff", vdiff testing involves taking snapshot images of the browser and comparing them against a known golden (or "baseline") image. The images are compared pixel-by-pixel and differences beyond a threshold will fail the test. Our vdiff testing leverages the [pixelmatch](https://github.com/mapbox/pixelmatch) library to perfom its comparison. +#### Options -Use the asynchronous `.to.be.golden()` Chai assertion to take a vdiff snapshot and compare it against its golden. +| Name | Type | Default | Description | +| :--- | :---: | :---: | :--- | +| group | `String` | `'test'` | Name of the test group to run | +| chrome | `Boolean` | `true` | Run tests in Chromium | +| firefox | `Boolean` | `true` | Run tests in Firefox | +| safari | `Boolean` | `true` | Run tests in Webkit | +| timeout | `Number` | `2000` | Test timeout threshold in ms | +| filter | `String` | | Filter test files by replacing wildcards with this glob | +| grep | `String` | | Only run tests matching this string or regexp | +| files | `String` | `'./test/**/*..js'` | Test files to run. Path or glob. | +| config | `String` | `'./d2l-test-runner.config.js'` | Location of config file | +| watch | `Boolean` | `false` | Reload tests on file changes. Allows debugging in all browsers. | +| open | `Boolean` | `false` | Open the browser in headed mode | +| slowmo | `Number` | | Slows down test operations by the specified number of milliseconds. Useful for debugging. | +| slow | `Number` | `75` | Tests whose duration in milliseconds are at most half of this threshold are "fast" and tests which exceed it are "slow" | + +For example, to run all tests in the default `'test'` group in Firefox and with a `3s` timeout: + +```bash +d2l-test-runner --firefox --timeout=3000 +``` + +A `d2l-test-runner.config.js` file in the current working directory (or the path from the `config` CLI flag) can also be used to configure `d2l-test-runner`. + +```javascript +export default { + firefox: true, + timeout: 3000 +}; +``` + +### Test Groups + +Tests are organized into groups, which can be configured and run together. + +The group name appears in the default `files` pattern (`'./test/**/*..js'`), making it typical for test files to have the group name as part of their extension. For example, the default group is `'test'` so all test files named `*.test.js` will belong to it by default. Similarly, the `vdiff` group contains files named `*.vdiff.js`. + +To run tests which match the pattern `'./test/**/*.mygroup.js'`: +```bash +d2l-test-runner --group mygroup +``` + +### Running a Subset of Tests + +While writing or debugging tests, it can be desirable to focus the runner on a subset of tests. + +### By File Name + +Use the `filter` option to filter by file name. It replaces any wildcards in the file name portion of the `files` pattern with the provided [glob](https://en.wikipedia.org/wiki/Glob_(programming)). + +For example, with the `'test'` group and default pattern `'./test/**/*..js'`, passing `d2l-test-runner --filter foo` will run tests which match `'./test/**/foo.test.js'`. + +Wildcards can still be used but need to be escaped. So `d2l-test-runner --filter foo\*` will run tests which match `'./test/**/foo*.test.js'`. + +### By Test Name + +Use the `grep` option to filter by test name. Only tests whose names match the provided string or regular expression will be run, regardless of file name. + +For example, `d2l-test-runner --grep foo` will run any test whose test suite(s) or name contains "foo". + +> **Note:** unfortunately, tests which do not match the grep value will be reported as failed instead of skipped. + +### Debugging Tests + +When tests don't go as expected, the next step is usually to debug them using the browser's built-in developer tools. + +There are two options for debugging: +* `watch`: after running `d2l-test-runner --watch`, choose "D" (debug in the browser) and select which test file to launch +* `open`: is a shortcut for `watch` that opens a browser window for each file sequentially. For large projects, use it in combination with `filter` to limit which file(s) are opened. For example: `d2l-test-runner --filter foo --watch`. + +With both `watch` and `open`, the browser debugger will be paused at the top of the file under test, which provides an opportunity to attach breakpoints if desired. A toolbar at the top of the screen then allows for each individual test to be skipped or run, as well as a "run all" option. + +## Vdiff Testing + +Short for "visual diff" and also known as "visual regression" or "perceptual diff", vdiff testing involves taking snapshot images of the browser and comparing them against a known golden (or "baseline") image. The images are compared pixel-by-pixel and differences beyond a threshold will fail the test. `@brightspace-ui/testing`'s vdiff leverages the [pixelmatch](https://github.com/mapbox/pixelmatch) library to perfom its comparison. + +### Writing Vdiff Tests + +Vdiff tests are written [just like other tests](#writing-tests), and the same utilities (`focusElem`, `oneEvent`, etc.) and `fixture` configuration options (viewport, language) are available. + +Use the asynchronous `.to.be.golden()` Chai assertion to take a vdiff snapshot and [compare it against its golden](#generating-the-goldens). ```javascript import { fixture, html } from '@brightspace-ui/testing'; @@ -335,38 +419,104 @@ describe('my-elem', () => { }); ``` -The filename and location of the resulting image will be based on the suite names and test name. +### Configuring the Snapshot Area -Sometimes you might be writing tests to verify a component's position within the whole viewport. For snapshots of the whole page, set up your components and pass `document` to the assertion: +By default, the snapshot area will be a rectangle around the target element plus a `10px` buffer margin on each side. To use a different margin, pass it as an option: ```javascript -import { fixture, html } from '@brightspace-ui/testing'; +await expect(elem).to.be.golden({ margin: 20 }); +``` -describe('my-elem', () => { - describe('situation1', () => { - it('state1', async() => { - await fixture(html``); - await expect(document).to.be.golden(); - }); - }); -}); +#### Capturing the Viewport + +To capture the entire viewport, pass `document` as the target element to the assertion: + +```javascript +await expect(document).to.be.golden(); +``` + +#### Changing the Vdiff Target + +If the target element doesn't accurately represent the area to be captured, the `vdiff-target` CSS class can be applied to the element that does. + +In the following example, the initial target element's area will not contain the nested element. By adding the `vdiff-target` CSS class to the nested element, it becomes the new vdiff target. + +```javascript +const elem = await fixture(html` + +
hello
+ +`); +await expect(elem).to.be.golden(); ``` -#### Configuring the Snapshot Area +#### Including Other Elements -By default, the snapshot area will be a rectangle around the source element plus a `10px` buffer margin on each side. To use a different margin, pass it as an option: +Elements using `absolute` or `fixed` positioning (like dropdowns or tooltips) may overflow the target element's capture area. To include them, apply the `vdiff-include` CSS class. + +In this example, the tooltip is positioned below the button and would not be captured. By applying `vdiff-include` to one or more elements, the captured area becomes the rectangle containing the initial target and all additional `vdiff-include` elements. ```javascript -await expect(elem).to.be.golden({ margin: 20 }); +const elem = await fixture(html` +