NEW e2e environment using playground cli

[carstingaxion]

This PR allows GatherPress to run automated & manual end-to-end tests, while sharing the same, wp-playground/cli powered, setup. The started playground imports the GatherPress/demo-data, that can be used instead of mocks or fixtures (for now).

NEEDED nice-to-have, before merge

• Update all blocks to API Version 3, to be able to use editor.canvas easily.
  (Feature/e2e tests workflow #776 (comment))

New for docs/developer

E2E Testing using Playwright and wp-now

GatherPress allows to run automated & manual end-to-end tests, while sharing the same, wp-playground/cli based, setup.

Automated tests

Check the results of the e2e-tests action workflow at https://github.com/GatherPress/gatherpress/actions/workflows/e2e-tests.yml.

Manual testing

Install dependencies

To run the E2E tests you will have to install playwright using the following command:

npx playwright install --with-deps

[!NOTE]
You also need to use Node.js 20 or later

Install the dependencies to create the testing instance, using the following command:

npm ci --legacy-peer-deps

Run the E2E tests

A call to npm run test:e2e... will automatically setup a wp-playground/cli powered WordPress instance.

The testing is website is reachable at http://127.0.0.1:9400, the user is admin and the password is password.

Choose one of the following options

1. For the headless mode, use the following command:

   npm run test:e2e

2. Run Playwright visually (to run tests in isolation and change what's happening), use:

   npm run test:e2e:ui

   

3. For debug mode (which will open the browser along with Playwright Editor and allows you to record what's happening), use the following command:

   npm run test:e2e:debug

   Run all the tests against a specific project.

   npm run test:e2e:debug -- project=webkit

   Run files that have events.spec in the file name.

   npm run test:e2e:debug -- events.spec

   [!NOTE]
   When writing a test, using the debug mode is recommended since it will allow you to see the browser and the test in action.

4. Run Tests independently AND visually using the Playwright VSCode extension

More about E2E testing

Start here:

• Playwright Documentation
• End-To-End Playwright test utils for WordPress

from the WordPress handbooks

• End-to-End Testing – Block Editor Handbook | Developer.WordPress.org
• Overusing snapshots – Block Editor Handbook | Developer.WordPress.org

More about wp-playground/cli, as the testing environment

Examples with great documentation:

• Playground CLI · WordPress/wordpress-playground#1289
• PoC: Run E2E tests with WP Playground · WordPress/gutenberg#62692
• Use WordPress Playground · swissspidy/wp-performance-action#173

To see more examples of E2E tests, check the Gutenberg repository: https://github.com/WordPress/gutenberg/tree/trunk/test/e2e

[!NOTE]
If you are out of ideas on who to test, check the Gutenberg repository. It has a lot of examples of E2E tests that you can use as a reference.

---

Closes #693

How to test the Change

1. Checkout this PR locally
2. Follow the before mentioned installation instructions
3. Run Playwright by one of the 4 given options, try npm run test:e2e for example

Changelog Entry

Changed environment for e2e-tests to run on wp-playground/cli

Credits

Props @carstingaxion

Checklist:

• I agree to follow this project's Code of Conduct.
• I have updated the documentation accordingly.
• I have added tests to cover my change.
• All new and existing tests pass.

[carstingaxion]

webkit is behaving strangly different.

[carstingaxion]

Some tests are still failing and I'm not sure, what is the reason for that.

But, as the quest was to find a working environment, I would call this to be done and open it up for reviews.

[carstingaxion]

Note

Ready for review does NOT mean ready for merge!

Just to note.

After fiddling with this workflow(s) for a while, now I need an outside view to don’t get crazy. I’m still unsure about the reasons for the timeouts.

[carstingaxion]

[adamziel]

If the database works for a while and then it breaks, It's likely this issue: WordPress/wordpress-playground#1596. I'd love to learn what happens there. Would you be able to log the first failing request and zip the playground site? It would also be supremely useful to peek into the /internal directory available through the PHP instance running inside the Playground CLI, but there's no easy API to do it from bash – we'd need a custom build that would zip it up and write to a local directory on the first error.

cc @WunderBart @swissspidy as these database error came up in your explorations, too.