Add option to partition results table by measurements #198

kevinpschaaf · 2020-10-09T10:19:32Z

Fixes #197

~~collate: false~~ partition: undefined:

~~collate: true~~ partition: "measurement":

kevinpschaaf · 2020-10-09T23:33:14Z

@aomarks I can't seem to get the Travis tests to re-run (it looks like a Safari flake in an unrelated test?)

aomarks

Looks good, just a comment about the name.

aomarks · 2020-10-10T00:55:09Z

CHANGELOG.md

@@ -7,6 +7,9 @@ project adheres to [Semantic Versioning](http://semver.org/).

 <!-- ## Unreleased -->

+- Add `collate` option for collating results tables by measurement (when


How about partition: "measurement"?

I also thought about names compare and group-by, but I'm leaning towards partition because it seems to best describe "the thing you want to do to the table after you see that it's too big".

Also I'm thinking the value should be measurement instead of true so that we have more room to expand with more partitioning options in the future.

WDYT?

Oh, that's nice. So conceptually we could add something like partition: "browser" to partition the results table based on browser? And next level, make it an array to partition on multiple dimensions, i.e. browser and measurement?

Sounds good. Was out yesterday but will try and get an update up soon.

I've addressed the feedback. All instances of collate: true have been replaced by partition: "measurement".
I also added documentation to the README for the new option.

andrewiggins

Love this! 🎉

andrewiggins · 2020-10-10T01:15:22Z

CHANGELOG.md

@@ -7,6 +7,9 @@ project adheres to [Semantic Versioning](http://semver.org/).

 <!-- ## Unreleased -->

+- Add `collate` option for collating results tables by measurement (when


Oh, that's nice. So conceptually we could add something like partition: "browser" to partition the results table based on browser? And next level, make it an array to partition on multiple dimensions, i.e. browser and measurement?

andrewiggins · 2020-10-10T01:16:17Z

package.json

@@ -13,6 +13,7 @@
  "scripts": {
    "prepare": "if [ -f './tsconfig.json' ]; then npm run build; fi;",
    "build": "rimraf lib/ client/lib/ && mkdir lib && npm run generate-json-schema && tsc && tsc -p client/ && npm run lint",
+    "build:watch": "tsc --watch",


andrewiggins · 2020-10-10T01:26:16Z

src/format.ts

@@ -104,6 +105,23 @@ export function automaticResultTable(results: ResultStats[]): AutomaticResults {
  return {fixed: fixedTable, unfixed: unfixedTable};
 }

+export function collatedResultTables(results: ResultStatsWithDifferences[]) {


Would there be any value in skipping the difference computation for dimensions that are partitioned? Though I guess that is larger breaking change that impacts more parts of the code.... Perhaps something to explore later?

marvinhagemeister · 2020-12-08T07:24:07Z

Love this! It is a huge improvement for readability of the results 🙌

marvinhagemeister · 2021-05-30T23:23:58Z

Still interested in having this. What are the open tasks to get this in?

prudepixie · 2022-06-24T17:06:53Z

Hi there, are there any plans to get this merged? Very useful and would like to use it in our benchmark set up

prudepixie · 2022-07-17T02:16:06Z

Hi there, are there any plans to get this merged? Very useful and would like to use it in our benchmark set up

@aomarks 🥇

…urements

src/config.ts

AndrewJakubowicz · 2023-11-30T00:09:57Z

I've found an edge case for small results tables that can't be partitioned. E.g. ./bin/tach.js --config lit-element/list/tachometer.json, where ./bin/tach.js is tachometer built from this PR. This returns:

┌─────────────┬─────────────────┐
│     Version │ <none>          │
├─────────────┼─────────────────┤
│     Browser │ chrome-headless │
│             │ 119.0.6045.199  │
├─────────────┼─────────────────┤
│ Sample size │ 50              │
├─────────────┼─────────────────┤
│       Bytes │ 30.95 KiB       │
└─────────────┴─────────────────┘

┌───────────────────┐
│          Avg time │
├───────────────────┤
│ 19.81ms - 20.64ms │
└───────────────────┘

┌─────────────────────┐
│            Avg time │
├─────────────────────┤
│ 202.71ms - 205.71ms │
└─────────────────────┘

┌─────────────────────┐
│            Avg time │
├─────────────────────┤
│ 196.53ms - 198.90ms │
└─────────────────────┘

This has been partitioned from:

┌──────────────────────────────┬─────────────────────┬─────────────────────────┬─────────────────────────┬─────────────────────────────────┐
│ Benchmark                    │            Avg time │ vs this-change [render] │ vs this-change [update] │ vs this-change [update-reflect] │
├──────────────────────────────┼─────────────────────┼─────────────────────────┼─────────────────────────┼─────────────────────────────────┤
│ this-change [render]         │   20.32ms - 21.24ms │                         │                  faster │                          faster │
│                              │                     │                -        │               90% - 90% │                       89% - 90% │
│                              │                     │                         │     183.28ms - 187.37ms │             177.64ms - 181.91ms │
├──────────────────────────────┼─────────────────────┼─────────────────────────┼─────────────────────────┼─────────────────────────────────┤
│ this-change [update]         │ 204.11ms - 208.10ms │                  slower │                         │                          slower │
│                              │                     │             868% - 916% │                -        │                         1% - 4% │
│                              │                     │     183.28ms - 187.37ms │                         │                 2.66ms - 8.43ms │
├──────────────────────────────┼─────────────────────┼─────────────────────────┼─────────────────────────┼─────────────────────────────────┤
│ this-change [update-reflect] │ 198.47ms - 202.64ms │                  slower │                  faster │                                 │
│                              │                     │             841% - 889% │                 1% - 4% │                        -        │
│                              │                     │     177.64ms - 181.91ms │         2.66ms - 8.43ms │                                 │
└──────────────────────────────┴─────────────────────┴─────────────────────────┴─────────────────────────┴─────────────────────────────────┘

My expectation would be that there would still be a benchmark name. On further investigation, this seems to be working as intended for small tables. However when partitioning a table into 3 small tables, the result becomes confusing due to a lack of benchmark name field.

kevinpschaaf added 2 commits October 9, 2020 03:10

Add option to collate results tables by measurements

4f57b1e

Add CHANGELOG

f9f2f91

google-cla bot added the cla: yes label Oct 9, 2020

kevinpschaaf added 5 commits October 9, 2020 12:22

Add collate flag to existing tests

49bdd44

Add collated table formatting tests

cd89838

Simplify

8c909ea

Use measurementName helper

a7e12e7

Format

c71f051

kevinpschaaf marked this pull request as ready for review October 9, 2020 23:29

kevinpschaaf requested a review from aomarks October 9, 2020 23:29

aomarks approved these changes Oct 10, 2020

View reviewed changes

andrewiggins reviewed Oct 10, 2020

View reviewed changes

AndrewJakubowicz added 4 commits November 27, 2023 14:03

Merge branch 'main' of github.com:google/tachometer into collate-meas…

c8d822d

…urements

fix build error

8a5d818

update tests so they all pass

97ae7fd

Replace collate option to use partition: "measurement"

9d779ec

AndrewJakubowicz changed the title ~~Add option to collate results tables by measurements~~ Add option to partition results table by measurements Nov 28, 2023

AndrewJakubowicz requested review from augustjk and aomarks November 28, 2023 04:44

fix typo in CHANGELOG

2bb8bcf

augustjk reviewed Nov 28, 2023

View reviewed changes

src/config.ts Outdated Show resolved Hide resolved

add "none" as the default value.

7d707cd

augustjk approved these changes Nov 28, 2023

View reviewed changes

AndrewJakubowicz requested a review from rictic November 28, 2023 23:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add option to partition results table by measurements #198

Add option to partition results table by measurements #198

kevinpschaaf commented Oct 9, 2020 •

edited by AndrewJakubowicz

Loading

kevinpschaaf commented Oct 9, 2020

aomarks left a comment

aomarks Oct 10, 2020

andrewiggins Oct 10, 2020

kevinpschaaf Oct 13, 2020

AndrewJakubowicz Nov 28, 2023

andrewiggins left a comment

andrewiggins Oct 10, 2020

andrewiggins Oct 10, 2020

andrewiggins Oct 10, 2020

marvinhagemeister commented Dec 8, 2020

marvinhagemeister commented May 30, 2021

prudepixie commented Jun 24, 2022

prudepixie commented Jul 17, 2022

AndrewJakubowicz commented Nov 30, 2023 •

edited

Loading

		@@ -7,6 +7,9 @@ project adheres to [Semantic Versioning](http://semver.org/).

		<!-- ## Unreleased -->

		- Add `collate` option for collating results tables by measurement (when

Add option to partition results table by measurements #198

Are you sure you want to change the base?

Add option to partition results table by measurements #198

Conversation

kevinpschaaf commented Oct 9, 2020 • edited by AndrewJakubowicz Loading

kevinpschaaf commented Oct 9, 2020

aomarks left a comment

Choose a reason for hiding this comment

aomarks Oct 10, 2020

Choose a reason for hiding this comment

andrewiggins Oct 10, 2020

Choose a reason for hiding this comment

kevinpschaaf Oct 13, 2020

Choose a reason for hiding this comment

AndrewJakubowicz Nov 28, 2023

Choose a reason for hiding this comment

andrewiggins left a comment

Choose a reason for hiding this comment

andrewiggins Oct 10, 2020

Choose a reason for hiding this comment

andrewiggins Oct 10, 2020

Choose a reason for hiding this comment

andrewiggins Oct 10, 2020

Choose a reason for hiding this comment

marvinhagemeister commented Dec 8, 2020

marvinhagemeister commented May 30, 2021

prudepixie commented Jun 24, 2022

prudepixie commented Jul 17, 2022

AndrewJakubowicz commented Nov 30, 2023 • edited Loading

kevinpschaaf commented Oct 9, 2020 •

edited by AndrewJakubowicz

Loading

AndrewJakubowicz commented Nov 30, 2023 •

edited

Loading