Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Record Playwright tests with Replay #348

Closed
wants to merge 14 commits into from
Closed

Record Playwright tests with Replay #348

wants to merge 14 commits into from

Conversation

bvaughn
Copy link
Owner

@bvaughn bvaughn commented May 1, 2024

No description provided.

Copy link

vercel bot commented May 1, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
react-resizable-panels ✅ Ready (Inspect) Visit Preview 💬 Add feedback May 19, 2024 1:50pm

package.json Outdated
@@ -23,6 +23,7 @@
"devDependencies": {
"@babel/preset-typescript": "^7.22.5",
"@playwright/test": "^1.37.0",
"@replayio/playwright": "3.0.0-alpha.3",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this setup work for you with this version now?

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. The latest comment I left in Linear mentioned that the steps are all missing. I'll add you to the workspace so you can look at them.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's see what happens with the newest version ( #349 ). I'm simultaneously testing this release in the overboard's repo

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm able to repro this locally using ur tests here... diving in! 🤿

@bvaughn
Copy link
Owner Author

bvaughn commented May 3, 2024

@Andarist The latest v6 alpha looks to have both steps and metadata. Great!

@bvaughn
Copy link
Owner Author

bvaughn commented May 3, 2024

Removing the test.only so that all of my tests run shows a new failure though:

RecordingUploadError: Failed to set recording metadata Error: Protocol client is not initialized

@Andarist
Copy link
Contributor

Andarist commented May 3, 2024

Interesting, gonna investigate soon!

package.json Outdated
@@ -23,6 +23,7 @@
"devDependencies": {
"@babel/preset-typescript": "^7.22.5",
"@playwright/test": "^1.37.0",
"@replayio/playwright": "3.0.0-alpha.6",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this could now be bumped to .7

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gotcha. Just bumped the version.

Noticed that I can't run e2e tests locally though:

$ p test:e2e        

> [email protected] test:e2e /Users/bvaughn/Documents/git/oss/react-resizable-panels/packages/react-resizable-panels-website
> playwright test

[replay.io]: Failed to send test metrics
[replay.io]: Error: request to https://webhooks.replay.io/api/metrics failed, reason: socket hang up
[replay.io]: 🕑 Completing some outstanding work ...
[replay.io]: 
[replay.io]: ❌ We encountered some unexpected errors processing your recordings and was unable to upload them.
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io

Maybe this is because I didn't pass the api token env var? But the error message isn't great if that's the case, and this would mean that no contributors could run these tests either.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also the terminal was totally silent for several seconds before printing this. I'm not sure if that meant tests weren't running, or if it was just swallowing the stdout output. (That output is important to debug.)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So you have this error, CI reports "RecordingUploadError: Failed to set recording metadata Error: Protocol client is not initialized" and I have "RecordingUploadError: Failed to set recording metadata Protocol error 1: Internal error" locally now 😂

Well, I'm investigating all of this now ;p

This one makes sure that the `@replayio/reply` (transitive dependency)
gets updated too (u can see the change in the lockfile).

With this one, I can record this repo and I get recordings. Some fail at
times when the metadata's size gets exceeded. We know about the issue
and we are in the process of figuring out the solution.
@Andarist
Copy link
Contributor

Andarist commented May 9, 2024

We have a partial success here! It still suffers from the metadata size problem - but most tests don't exceed that size and recordings are nicely created. Could you also confirm that it works for you locally?

@bvaughn
Copy link
Owner Author

bvaughn commented May 9, 2024

Could you also confirm that it works for you locally?

Yes, tests do run (although tests/StackingOrder.spec seems perhaps a bit flakier? could be a coincidence or could be a difference caused by the older Chromium version).

I think our runner's output is less helpful as you go. For instance, running these tests on main gives feedback as you go:
Screenshot 2024-05-09 at 10 26 40 AM

Running them on this branch is more quiet:
Screenshot 2024-05-09 at 10 27 16 AM

But that's a nit.

This addresses the complaint from
#348 (comment)
. I didn't realize that the default is `'list'` locally, I added
`'line'` reporter since I've seen that being used in a couple of other
projects. I totally agree that `'list'` is quite nice and I prefer it to
`'line'` - especially if we consider smaller test suites.

⚠️ `@replayio/playwright` still spits out a bunch of junk into the
stdout. That makes the current output very cluttered - even with this
improvement. This is going to be addressed with PRO-315
@bvaughn bvaughn closed this Jul 12, 2024
@bvaughn bvaughn deleted the replayio branch July 12, 2024 18:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants