E2E: update tests #1217

alexkuc · 2023-07-05T13:21:51Z

Need to update E2E tests to make them compatible with Playwright runner

Primary TODOs

Events Page - `events-page.spec.ts`

Categories - #1229

admin/event/categories/editCategory.test.ts
admin/event/categories/categoryShortcode.test.ts
admin/event/categories/createCategory.test.ts

Default Settings - #1230

admin/event/default/settings/defaultMaximumTickets.test.ts
admin/event/default/settings/defaultRegistrationStatus.test.ts

Links - #1236

admin/event/events-list/linkForDraft.test.ts
admin/event/events-list/linkForToday.test.ts
admin/event/events-list/linkForViewAllEvents.test.ts
admin/event/events-list/linkForThisMonth.test.ts
admin/event/events-list/linkForTrash.test.ts

Single Event - `single-event.spec.ts` - #1238

shared/pagination.test.ts
shared/search.test.ts
shared/date-capacity-vs-ticket-qty.test.ts
shared/views/switch-view.test.ts
shared/views/card-view-date-range.test.ts
shared/copy-entity.test.ts

Registration - `registration.spec.ts`

shared/two-dates-two-tickets.test.ts
public/single-page-checkout/registration-scenarios/sold-out.test.ts
public/single-page-checkout/registration-scenarios/index.test.ts

Add-ons - `rem.spec.ts`, `wpUser.spec.ts`

add-ons/rem/index.test.ts
add-ons/wpUser/index.test.ts

To Be Sorted ⚠️

Configuration

~~show only PHP errors but suppress warnings~~ (moved to #1234)

Secondary TODOs (optional)

make docker-related information like HOST or PORT configurable instead of being hard-coded (ref) Update Playwright configuration for E2E tests #1228
make credentials non hardcoded Update Playwright configuration for E2E tests #1228
create E2E for verifying (testing) environment is functional and works as expected (sanity test) Update Playwright configuration for E2E tests #1228 (environment.spec.ts)
remove deprecated scripts from package.json Remove dev dependencies for old E2E tests #1231
remove dependencies mentioned in Chore: review Playwright config #1213 Remove dev dependencies for old E2E tests #1231
move dependencies from root workspace to package @eventespress/e2e Consolidate E2E deps under its own package #1243

The text was updated successfully, but these errors were encountered:

alexkuc · 2023-07-05T22:46:48Z

Consider removal of packages mentioned in #1213

alexkuc · 2023-07-07T00:15:28Z

Consider moving dependencies from root workspace to @eventespress/e2e

alexkuc · 2023-07-07T00:27:41Z

Until #1223 is merged, created this branch from commit 2069bb624412823030bac3a691cc181dec61b2a0

Edit: no need for rebase, PR #1223 was merged and didn't commit anything yet

alexkuc · 2023-07-07T01:51:50Z

To implement flexible support for values like HOST or PORT, consider use of ts-node to use TypeScript for make-config (instead of plain JS to leverage intellisense) and this issue on how to get it working

alexkuc · 2023-07-07T02:08:48Z

Once tests are converted, remove deprecated scripts from package.json

alexkuc · 2023-07-07T18:08:30Z

Bootstrap code from legacy E2E tests that runs before every suite from setup-playwright.ts

// Before every test suite run, delete all content created by the test. This ensures
// other posts/comments/etc. aren't dirtying tests and tests don't depend on
// each other's side-effects.
beforeAll(async () => {
	await loginUser(); // log in into /wp-admin/wp-login.php to obtain auth cookies

        // IS_WP_MULTISITE_NETWORK === true => run E2E against http://e2etests.makeeventsnotwar.com
        // IS_WP_MULTISITE_NETWORK === false => run E2E against docker (@wordpress/env)
	if(!IS_WP_MULTISITE_NETWORK){
		await activatePlugin('event-espresso-core/espresso.php'); // go to /wp-admin/plugins.php and activate plugin
		await activatePlugin('barista/ee-barista.php');  // go to /wp-admin/plugins.php and activate plugin	
	}
	
	await Goto.eventsListPage(); // go to /wp-admin/admin.php?page=espresso_events
	await defaultSettingsManager.gotoDefaultSettings(); // go to tab "Default Settings" in EE's admin page
        // choose setting for "Activate Advanced Editor"
        // 1 => Advanced Editor
        // 0 => Legacy Editor
	await defaultSettingsManager.selectDefaultEditor('1');
});

afterAll(async () => {
        // IS_WP_MULTISITE_NETWORK === true => run E2E against http://e2etests.makeeventsnotwar.com
        // IS_WP_MULTISITE_NETWORK === false => run E2E against docker (@wordpress/env)
	if(!IS_WP_MULTISITE_NETWORK){
		await deactivatePlugin('barista/ee-barista.php');  // go to /wp-admin/plugins.php and deactivate plugin
	}
});

alexkuc · 2023-07-07T18:54:21Z

💡 Create a E2E to confirm it is configured and functional?

alexkuc · 2023-07-11T08:35:23Z

Fixture for activating/deactivating plugin has proven to be flaky i.e. intermittent failures when running repeat tests. Will proceed with the following solution

use global setup for plugin activation/deactivation
implement provisioning of unique WordPress credentials per worker

alexkuc · 2023-07-12T01:52:23Z

Investigate why opening Categories fails:

alexkuc · 2023-07-12T02:20:43Z

Something overwrites WP_DEBUG to false

Edit: overwrite comes from @wordpress/env, see default wp-config.php values

alexkuc · 2023-07-12T02:51:40Z

Looks like the issue was solved after running yarn docker:stop && yarn docker:start --update i.e. looks like it was broken/dirty state of the environment. Fyi, running just yarn docker:start --update is sufficient as @wordpress/env is smart enough to stop docker container(s) first.

alexkuc · 2023-07-12T08:22:25Z

Base for actual E2E branches - 78b59fbd9688938dae34a44156e1f0b058d6c9a7 - until PR #1228 is merged into master

alexkuc · 2023-07-12T21:50:22Z

@tn3rb Just FYI, I know this issue seemed like it was in limbo but actually lots of work was being done, I just didn't update TODOs on time

alexkuc · 2023-07-12T22:06:46Z

@tn3rb Just FYI, when converting tests, I am following the testing scope of whatever was in the original E2E tests. This is to avoid scope creep. My vision is to convert existing E2E tests and hook them up as a part of our pipeline asap. Then we can open subsequent GH issues and/or PRs to improve those tests. I see that there is a lot of great value for the company by automating E2E so I'd rather follow incremental Agile approach instead of trying to get "picture perfect" E2E tests (waterfall). However… if you prefer to follow a different approach, please let me know and I'll adapt accordingly.

alexkuc · 2023-07-13T00:08:03Z

Since #1228 is merged, all E2E branches are to be based off master branch

alexkuc · 2023-07-13T09:03:06Z

Experiencing intermittent failures again…

Error: page.reload: Navigation failed because page was closed!
=========================== logs ===========================
waiting for navigation until "load"
  navigated to "http://<host>:<port>/wp-admin/admin.php?page=espresso_events&action=create_new"
============================================================

  118 |
  119 | 			// reload page to check if default settings have applied
> 120 | 			await eventPage.reload();
      | 			                ^
  121 |
  122 | 			await expect(eventDropdown).toHaveValue(value);
  123 |

    at /<path>/events-page.spec.ts:120:20

alexkuc · 2023-07-13T09:18:29Z

During investigation I have noticed that auth cookies are missing

alexkuc · 2023-07-13T09:29:31Z

Looks like increasing timeout per test to 60 seconds resolved the issue

alexkuc · 2023-07-13T09:49:36Z

I need to consider that not all E2E tests will support concurrency, such as testing global settings

alexkuc · 2023-07-13T10:05:20Z

The alternative to run tests in series is to actually support multiple databases. Consider official WordPress docker image.

alexkuc · 2023-07-13T10:09:31Z

Before proceeding anyway, let's do a quick investigation to confirm this theory that running tests sequentially will resolve issue of having unexpected values in assert conditions

alexkuc · 2023-07-13T10:27:43Z

Thinking more about… having a separate database instance per work might be way too hard to implement since that would require some kind of broker mechanism to spin up new WordPress instance and provision new database.

alexkuc · 2023-07-13T10:29:06Z

By the way, running tests sequentially leads to no errors but timeframe is increased by x5-10

alexkuc · 2023-07-14T10:10:17Z

Investigating flakiness

Runs

Command	Outcome
`yarn e2e:run --repeat-each 3 --project chromium`	✅
`yarn e2e:run --repeat-each 3 --project chromium --workers 1`	✅
`yarn e2e:run --repeat-each 3 --project chromium --workers 1 --trace on`	✅
`yarn e2e:run --repeat-each 4 --project chromium --workers 1`	❌
`yarn e2e:run --repeat-each 4 --project chromium --workers 1 --trace on`	❌
`yarn e2e:run --repeat-each 5 --project chromium`	❌

Note: test case is marked as test.only

alexkuc · 2023-07-17T10:59:01Z

When running tests in series, it appears that fixture Auth fails as the test times out on the login page when attempting to navigate to admin page for EE events

Edit: Strangely enough, it is the initial/admin tab that fails to re-use auth cookie cookie. We actually expect to re-use same admin auth session. Similar, to how it is possible to open several WP admin pages under the same account.

alexkuc · 2023-07-17T11:09:02Z

When the Auth fixture fails, it looks like auth cookie is wrong as it has value of

Cookie: wordpress_test_cookie=WP%20Cookie%20check

alexkuc · 2023-07-17T11:45:40Z

It appears that even when auth cookie is present, test is getting timeout at the login page. What I have noticed is that redirect is indeed happening due to WordPress. Curiously enough, there is search param reauth=1 present for some reason.

alexkuc · 2023-07-17T14:01:05Z

Once I have enabled screenshots and trace on Auth fixture, the error seems gone?

await page.context().tracing.start({
	screenshots: true,
	snapshots: true,
	sources: true,
});

await page.screenshot({
  fullPage: true,
  path: `<repo>/debug-ss/ss-${this.workerInfo.workerIndex}-${this.workerInfo.parallelIndex}.png`,
});

await page.context().tracing.stop({
  path: `<repo>/debug-trace/trace-${this.workerInfo.workerIndex}-${this.workerInfo.parallelIndex}.zip`,
});

alexkuc · 2023-07-18T17:35:32Z

To avoid issues with "floating" time, I need to set it to something fixed and used static timestamps to avoid unexpected surprises. Consider this solution.

alexkuc · 2023-07-20T20:26:17Z

As per my conversation with @tn3rb instead of doing implicit checks, e.g. count how many datetimes an event has, I should explicitly checks parameter, e.g. does a specific datetime exist for a given event?

alexkuc · 2023-07-21T18:13:36Z

Scaffolding data via UI fixtures proved unreliable. I will investigate REST API route for scaffolding data.

alexkuc · 2023-07-27T19:34:56Z

Error reporting constants

E_ERROR
E_PARSE
E_CORE_ERROR
E_COMPILE_ERROR
E_USER_ERROR
E_RECOVERABLE_ERROR

from https://www.php.net/manual/en/errorfunc.constants.php

Bitmask value for error_reporting() is 4437 thanks to this calculator

alexkuc · 2024-02-18T11:18:15Z

Closing this in favour of ~~https://github.com/eventespresso/e2e-tests/issues/31~~ https://github.com/eventespresso/e2e-tests/issues/74

alexkuc · 2024-03-10T17:25:11Z

Files references in this issue can be found in this PR or via file browser here

alexkuc added D: Packages 📦 domain: Barista Packages P3: med priority 😐 priority: medium C: tests 🧪 category S1: new 👶🏻 status labels Jul 5, 2023

alexkuc self-assigned this Jul 5, 2023

This was referenced Jul 6, 2023

Chore: review Playwright config #1213

Closed

Fix/e2e tests #1223

Merged

alexkuc mentioned this issue Jul 12, 2023

Update Playwright configuration for E2E tests #1228

Merged

alexkuc mentioned this issue Jul 12, 2023

Create Events page Categories E2E test #1229

Merged

alexkuc mentioned this issue Jul 12, 2023

Add Default Settings E2E test to Events page E2E test #1230

Merged

alexkuc mentioned this issue Jul 13, 2023

Remove dev dependencies for old E2E tests #1231

Merged

alexkuc added the T: chore 🧹 type: REPO - maintenance label Jul 18, 2023

This was referenced Jul 18, 2023

Create E2E test for Quick Links in Events page #1235

Closed

Create E2E test for Quick Links in Events page #1236

Merged

alexkuc mentioned this issue Jul 20, 2023

Create E2E test for single Event page #1238

Closed

alexkuc mentioned this issue Jul 27, 2023

E2E: allow parallel execution of tests #1234

Closed

alexkuc mentioned this issue Aug 4, 2023

E2E: set error_reporting #1248

Closed

alexkuc closed this as completed Feb 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

E2E: update tests #1217

E2E: update tests #1217

alexkuc commented Jul 5, 2023 •

edited

Loading

alexkuc commented Jul 5, 2023

alexkuc commented Jul 7, 2023

alexkuc commented Jul 7, 2023 •

edited

Loading

alexkuc commented Jul 7, 2023

alexkuc commented Jul 7, 2023

alexkuc commented Jul 7, 2023 •

edited

Loading

alexkuc commented Jul 7, 2023

alexkuc commented Jul 11, 2023

alexkuc commented Jul 12, 2023

alexkuc commented Jul 12, 2023 •

edited

Loading

alexkuc commented Jul 12, 2023

alexkuc commented Jul 12, 2023 •

edited

Loading

alexkuc commented Jul 12, 2023

alexkuc commented Jul 12, 2023 •

edited

Loading

alexkuc commented Jul 13, 2023

alexkuc commented Jul 13, 2023

alexkuc commented Jul 13, 2023

alexkuc commented Jul 13, 2023

alexkuc commented Jul 13, 2023

alexkuc commented Jul 13, 2023

alexkuc commented Jul 13, 2023

alexkuc commented Jul 13, 2023

alexkuc commented Jul 13, 2023

alexkuc commented Jul 14, 2023 •

edited

Loading

alexkuc commented Jul 17, 2023 •

edited

Loading

alexkuc commented Jul 17, 2023

alexkuc commented Jul 17, 2023

alexkuc commented Jul 17, 2023

alexkuc commented Jul 18, 2023

alexkuc commented Jul 20, 2023

alexkuc commented Jul 21, 2023

alexkuc commented Jul 27, 2023

alexkuc commented Feb 18, 2024 •

edited

Loading

alexkuc commented Mar 10, 2024

E2E: update tests #1217

E2E: update tests #1217

Comments

alexkuc commented Jul 5, 2023 • edited Loading

Primary TODOs

Events Page - events-page.spec.ts

Categories - #1229

Default Settings - #1230

Links - #1236

Single Event - single-event.spec.ts - #1238

Registration - registration.spec.ts

Add-ons - rem.spec.ts, wpUser.spec.ts

To Be Sorted ⚠️

Configuration

Secondary TODOs (optional)

alexkuc commented Jul 5, 2023

alexkuc commented Jul 7, 2023

alexkuc commented Jul 7, 2023 • edited Loading

alexkuc commented Jul 7, 2023

alexkuc commented Jul 7, 2023

alexkuc commented Jul 7, 2023 • edited Loading

alexkuc commented Jul 7, 2023

alexkuc commented Jul 11, 2023

alexkuc commented Jul 12, 2023

alexkuc commented Jul 12, 2023 • edited Loading

alexkuc commented Jul 12, 2023

alexkuc commented Jul 12, 2023 • edited Loading

alexkuc commented Jul 12, 2023

alexkuc commented Jul 12, 2023 • edited Loading

alexkuc commented Jul 13, 2023

alexkuc commented Jul 13, 2023

alexkuc commented Jul 13, 2023

alexkuc commented Jul 13, 2023

alexkuc commented Jul 13, 2023

alexkuc commented Jul 13, 2023

alexkuc commented Jul 13, 2023

alexkuc commented Jul 13, 2023

alexkuc commented Jul 13, 2023

alexkuc commented Jul 14, 2023 • edited Loading

alexkuc commented Jul 17, 2023 • edited Loading

alexkuc commented Jul 17, 2023

alexkuc commented Jul 17, 2023

alexkuc commented Jul 17, 2023

alexkuc commented Jul 18, 2023

alexkuc commented Jul 20, 2023

alexkuc commented Jul 21, 2023

alexkuc commented Jul 27, 2023

alexkuc commented Feb 18, 2024 • edited Loading

alexkuc commented Mar 10, 2024

alexkuc commented Jul 5, 2023 •

edited

Loading

Events Page - `events-page.spec.ts`

Single Event - `single-event.spec.ts` - #1238

Registration - `registration.spec.ts`

Add-ons - `rem.spec.ts`, `wpUser.spec.ts`

alexkuc commented Jul 7, 2023 •

edited

Loading

alexkuc commented Jul 7, 2023 •

edited

Loading

alexkuc commented Jul 12, 2023 •

edited

Loading

alexkuc commented Jul 12, 2023 •

edited

Loading

alexkuc commented Jul 12, 2023 •

edited

Loading

alexkuc commented Jul 14, 2023 •

edited

Loading

alexkuc commented Jul 17, 2023 •

edited

Loading

alexkuc commented Feb 18, 2024 •

edited

Loading