Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky E2E Metrics App Start Time #3413

Open
krystofwoldrich opened this issue Nov 22, 2023 · 4 comments · May be fixed by #4292
Open

Flaky E2E Metrics App Start Time #3413

krystofwoldrich opened this issue Nov 22, 2023 · 4 comments · May be fixed by #4292
Assignees
@antonis
Labels
Platform: React-Native repository maintenance

Comments

@krystofwoldrich
Copy link
Member

krystofwoldrich commented Nov 22, 2023 •
edited
Loading

Description

Can we improve the app start time measurement tests?

Often the test fails (looking at the last 30 commits on main cca 1/3 where only metrics failed) because the App With Sentry started faster.

https://github.com/getsentry/sentry-react-native/actions/runs/6710776019/job/18236705951
@krystofwoldrich
Copy link
Member Author

@vaind I've seen there are some conditions in place to avoid this, is there something else that we could improve?

@krystofwoldrich
Copy link
Member Author

Recently I also noticed this

https://github.com/getsentry/sentry-react-native/actions/runs/6954144218/job/18920431622

StartupTimeTest > startup time() FAILED
    org.openqa.selenium.WebDriverException: An unknown error has occurred
    Build info: version: '4.2.1', revision: 'ac4d0fdd4a'
    System info: host: 'Mac-1700637830941.local', ip: '10.213.5.236', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '12.7.1', java.version: '11.0.21'
    Driver info: io.appium.java_client.ios.IOSDriver
    Command: [744b0c72-5b9d-4a04-b3a2-c9f0a4490f4c, terminateApp {bundleId=io.sentry.rn.perf-test-app-sentry}]
    Capabilities {appium:appiumData: {appiumVersion: 2.0.0, automationName: XCUITest}, appium:automationName: XCUITest, appium:chromeOptions: {w3c: false}, appium:commandTimeouts: {default: 60000}, appium:databaseEnabled: false, appium:deviceContextId: b77f8d23-2f05-4e32-81a8-d89..., appium:deviceName: 00008101-000559D20203001E, appium:disableWindowAnimation: true, appium:javascriptEnabled: true, appium:jobUuid: b68759cda36f451eb12b6484163..., appium:locationContextEnabled: false, appium:networkConnectionEnabled: false, appium:orientation: PORTRAIT, appium:platformVersion: 14.8, appium:takesScreenshot: true, appium:testobject_device: iPhone_12_Pro_Max_real_us, appium:testobject_device_name: iPhone 12 Pro Max, appium:testobject_device_session_id: 926ee55c-d870-4465-b171-dc5..., appium:testobject_test_report_api_url: https://api.us-west-1.sauce.../, appium:testobject_test_report_url: https://app.saucelabs.com/t..., appium:testobject_user_id: sentryio, appium:udid: 00008101-000559D20203001E, appium:usedCachedDevice: false, appium:webDriverAgentUrl: http://127.0.0.1:5720/, appium:webStorageEnabled: false, browserName: safari, platformName: IOS, sauce:options: {appiumVersion: 2.0.0, build: CI getsentry/sentry-react-n..., name: Performance tests, realDevice: true, tags: [ios, ci]}}
    Session ID: 744b0c72-5b9d-4a04-b3a2-c9f0a4490f4c
        at [email protected]/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at [email protected]/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at [email protected]/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at [email protected]/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
        at app//org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.createException(W3CHttpResponseCodec.java:200)
        at app//org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:133)
        at app//org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:53)
        at app//org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:184)
        at app//io.appium.java_client.remote.AppiumCommandExecutor.execute(AppiumCommandExecutor.java:180)
        at app//org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:551)
        at app//org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:606)
        at app//io.appium.java_client.AppiumDriver.execute(AppiumDriver.java:[262](https://github.com/getsentry/sentry-react-native/actions/runs/6954144218/job/18920431622#step:14:267))
        at app//io.appium.java_client.CommandExecutionHelper.execute(CommandExecutionHelper.java:27)
        at app//io.appium.java_client.InteractsWithApps.terminateApp(InteractsWithApps.java:179)
        at app//io.appium.java_client.InteractsWithApps.terminateApp(InteractsWithApps.java:163)
        at app//StartupTimeTest.collectAppStartupTimes(StartupTimeTest.kt:194)
        at app//StartupTimeTest.access$collectAppStartupTimes(StartupTimeTest.kt:16)
        at app//StartupTimeTest$collectStartupTimes$1.invoke(StartupTimeTest.kt:114)
        at app//StartupTimeTest$collectStartupTimes$1.invoke(StartupTimeTest.kt:100)
        at app//TestBase.withDriver(TestBase.kt:26)
        at app//StartupTimeTest.collectStartupTimes(StartupTimeTest.kt:100)
        at app//StartupTimeTest.startup time(StartupTimeTest.kt:44)

@krystofwoldrich
Copy link
Member Author

We could ignore the invalid results. And not mark the job as failed.

@krystofwoldrich krystofwoldrich moved this from Needs Discussion to Todo in Mobile & Cross Platform SDK Dec 1, 2023
@krystofwoldrich krystofwoldrich moved this from Todo to Backlog in Mobile & Cross Platform SDK Jan 16, 2024
@krystofwoldrich
Copy link
Member Author

We are not measuring the app start correctly.

https://github.com/getsentry/action-app-sdk-overhead-metrics/blob/9615bad7552f3f34b12b433b5c31ea08acf7c301/src/test/kotlin/StartupTimeTest.kt#L223

The main activity of the application will be displayed before RN start executing and thus before the JS SDK inits.

@antonis antonis mentioned this issue Oct 9, 2024
Merged
9 tasks
@antonis antonis self-assigned this Nov 19, 2024
@antonis antonis moved this from Backlog to In Progress in Mobile & Cross Platform SDK Nov 19, 2024
@antonis antonis linked a pull request Nov 19, 2024 that will close this issue
Draft
9 tasks
@antonis antonis moved this from In Progress to Todo in Mobile & Cross Platform SDK Nov 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@antonis antonis

Labels
Platform: React-Native repository maintenance
Projects
Mobile & Cross Platform SDK
Status: Todo
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(test): Flaky E2E Android Metrics App Start Time getsentry/sentry-react-native
2 participants
@antonis @krystofwoldrich