puppeteer-layout-shifts

This repository contains some Node.js/Puppeteer scripts which automate measuring layout shifts on a large number of pages. They're currently pretty thrown together, but if they're useful they may evolve into a more polished tool.

puppeteer_cls.js

Usage: node puppeteer_cls.js <numRuns> <outDir> <chromePath> <urlFile>

• numRuns: number of times to run the page to gather CLS. The script will perform an additional run to gather a trace with debug layout shift info. This run won't be counted in the results in case the tracing affects the page shifts.
• outDir: output directory. The script generates url.txt and url.json for each URL into this output directory.
  • url.txt contains newline-separated results
  • url.json contains the trace of the run with debug layout shifts
• chromePath is the path to the Chrome executable. This allows the script to be used with a custom build of chromium.
• urlFile is a newline-separated list of urls.

This script runs the specified build of Chrome on the specified list of URLs, repeating the specified number of times, and calculates the layout shift for each run in JavaScript using the LayoutStability API. It outputs a .txt file with newline-separated CLS score results for each run into the specified output directory. Then it runs again to collect a trace with debug layout shift info, which it also outputs to the specified directory.

It runs devtools mobile emulation as a Nexus 5 with 4G network.

analyze_results.js

Usage: node analyze_results.js <urlFile> <outFile> <rev1> <rev2>... This script generates a CSV file from the output of the previous script. Its arguments are:

• urlFile: path to a newline-separated list of urls. Should be the same list passed into the previous script.
• outFile: path to CSV file that will be generated by this script
• rev1, rev2...: paired list of output directories from the previous script. These represent output before/after a change. The script can take any number of pairs. If these are not specified, the script will search the current directory for directories starting with 'r'.

For each pair of directories, the script looks for a .txt file for each url, reads this to get the results for that run, and compares the results with the same url for the other run to see if they are statistically significant, and if so whether there was a regression or progression. It writes out a row like this to the CSV file for each url:

url, rev1 median, rev2 median, result

Result can be one of the following:

• REGRESSION means the values increased in a statistically significant way between runs.
• IMPROVEMENT means the values decreased in a statistically significant way between runs.
• NO CHANGE means that that change between runs was not statistically significant (within the noise)
• UNKNOWN means that not enough values were generated for one or both runs for a comparison to be made.

Note that CLS scores on some sites are noisy, and so it is possible that results may appear statistically significant when they are not. Running with either a large set of URLs, a large number of repeats, or URLs known to have stable CLS values can alleviate this issue.