The purpose of this repository is to better understand the variance that occurs when running userflow.
The purpose of this experiment is to document the impact of Client hardware variability and Client resource contention when running user-flow on a GitHub Actions Runner. For this reason we attempted to mitigate the impact of the main sources of variance identified by lighthouse.
Mitigation Strategies
- Page Nondeterminism: Ensuring that the exact same version of the page is being tested between different runs.
- Local Network Variability: Serve application locally with no external resources.
- Tier-1 Network Variability: Serve application locally with no external resources.
- Web Server Variability: Serve application locally with no external resources.
- Browser Nondeterminism: Serve simple application with no complex logic.
Target Details
Application
The Target application is Coffee cart which is served locally using vite preview.
User Flow Run
The user was generated using npx @push-based/user-flow init and executed as a cron job in a GitHub CI workflow using npx @push-based/user-flow init over a GitHub runner.
Data Collected
The was collected between TODO and TODO during this time user-flow collect was executed 200 times inside a GitHub runner.
The user-flow only collected data from the performance category for an initial navigation.
After each run the fetch time and performance score we appended to the previews results and the rest of the json was deleted.
The results varied, with the highest score of 0.98, the lowest score of 0.87, a mean of 0.948 and a medium of 0.96.
