Async & Performance Chapter 6 "Repetition" section

[doubleOrt]

Yes, I promise I've read the Contributions Guidelines (please feel free to remove this line).

I've been having lots of trouble with 3 paragraphs in this section, IMO, the whole section is too dense and short.

This paragraph:

A straight mathematical average by itself is definitely not sufficient for making judgments about performance which you plan to extrapolate to the breadth of your entire application. With a hundred iterations, even a couple of outliers (high or low) can skew the average, and then when you apply that conclusion repeatedly, you even further inflate the skew beyond credulity.

1. Is the usage of the word "credulity" here a typo ? "credulity" is a synonym of "naivety" and "gullibility", I think the word is supposed to be "credibility" ?

2. What does "apply that conclusion repeatedly" really mean ? Does it mean "when you increase the number of iterations, the skew will increase proportionally" ?

The next paragraph:

Instead of just running for a fixed number of iterations, you can instead choose to run the loop of tests until a certain amount of time has passed. That might be more reliable, but how do you decide how long to run? You might guess that it should be some multiple of how long your operation should take to run once. Wrong.

1. Why might this be more reliable ? In the previous section, it is established that there are 3 main problems with a simple "startTime, do operation, endTime - startTime" benchmark:
   A) Something could have intervened with the engine or system during that specific test.
   B) The engine could have found a way to optimize your isolated case.
   C) Perhaps your timer was not precise enough, therefore you got an inaccurate result.

Using the second pattern, I believe problem A is still there, something could intervene with the engine causing an iteration of the test to take longer, therefore affecting the end result. I can't see why the engine can't find a way to optimize your tests with this pattern as well if it could do so with the first pattern, how much is it really different anyways ?
The second problem could solve problem C, but only if the time to repeat across is a multiple of the timer's precision (e.g 30ms for a timer with 15ms precision). Otherwise you would have to increase the time to repeat across to improve accuracy, and couldn't this be achieved by increasing the number of iterations in pattern A as well ?


And then the next paragraph:

Actually, the length of time to repeat across should be based on the accuracy of the timer you're using, specifically to minimize the chances of inaccuracy. The less precise your timer, the longer you need to run to make sure you've minimized the error percentage. A 15ms timer is pretty bad for accurate benchmarking; to minimize its uncertainty (aka "error rate") to less than 1%, you need to run your each cycle of test iterations for 750ms. A 1ms timer only needs a cycle to run for 50ms to get the same confidence.

I wonder what the maths behind this part is:> A 15ms timer is pretty bad for accurate benchmarking; to minimize its uncertainty (aka "error rate") to less than 1%, you need to run your each cycle of test iterations for 750ms. A 1ms timer only needs a cycle to run for 50ms to get the same confidence.

My brain is roaming in uncharted territory, so please excuse me if I am saying things that don't make any sense, but trust me, I have tried to comprehend this section for quite a while and this is my best so far, and it is all purely theoretical. I understand the basic message of this part of the book: "Use a library to benchmark your JavaScript code, it isn't as easy as it looks".

[getify]

Is the usage of the word "credulity" here a typo ? "credulity" is a synonym of "naivety" and "gullibility", I think the word is supposed to be "credibility" ?

I've learned some stuff as a result of you asking this question, so thanks!

1. Yes, I indeed was meaning "credibility". I incorrectly was believing that "creduilty" was basically a synonym of credibility.

2. In particular, I was using the phrase "beyond credulity" as a modified form of the more common and accepted "strains credulity". I was thinking "strains credulity" means "is very hard to believe is credible".

3. In fact, "strains credulity" means "is so hard to believe that even a gullible person would struggle to believe it."

4. In that sense, "beyond credulity" or "strains credulity" actually could fit with what I'm saying, but make the point even stronger than I was intending to make originally. I'm saying that if you do this thing -- taking a micro measurement that itself has a lot of doubt/skew to it -- and then repeat it a bunch across a wide section of your code, the results you thought you could trust in the small are now so unbelievable and not-credible, that it's silly... and indeed, even a gullible person would be unlikely to believe that those results were credible.

5. Though I'm stretching a bit to post-justify what I originally wrote, I think that point is valid and stands. However, the fact that I need to explain it in this much detail means it's more artistic writing and not clear technical writing. I think my editor @bmacdonald-editor would agree I should reword that part to be more clear and less embellished. So at a minimum, I'll keep this in mind for second edition revisions.

[getify]

What does "apply that conclusion repeatedly" really mean ? Does it mean "when you increase the number of iterations, the skew will increase proportionally" ?

It means use the same technique everywhere, assuming that in all cases the net effect on that larger scale will always follow directly from the smaller narrow observed effect from the single benchmark.

[getify]

I can't see why the engine can't find a way to optimize your tests with this pattern as well if it could do so with the first pattern, how much is it really different anyways ?

It certainly can happen, but it's statistically less likely to happen if you have a longer amount of time that a test may be being repeated across. Even those fluctuations get averaged out a bit. And if it's a fixed amount of time that establishes a statistical likelihood of some confidence level, then you know (mathematically) that it's more reliable to trust.

couldn't this be achieved by improving the number of iterations in pattern A as well ?

The point being made is that simply increasing the number of iterations isn't the most mathematically (statistically) sound way to increase the confidence of a result. Rather, the amount of time for the test to run is the better variable.

You should understand that I'm merely recounting perspectives from others' work on this topic, not trying to precisely and mathematically lay out a case. Their authority, and rigorous work, is the authority here, not my (in)ability to prove the case in my writing.

[getify]

I wonder what the maths behind this part is

Much of my observations in my text about the math/statistics behind these techniques is coming from the work @jdalton (and others) did on Benchmark.js.

I can't quite find/cite the information I used in my writing -- some blog post somewhere, I'm sure -- but actually their source code itself has a number of such citations behind the math/statistics. There are several such comments in there. Hopefully those will help if you want to explore the math deeper.

[doubleOrt]

I couldn't agree more about an edit to that specific paragraph (or section), both vocabulary edits and perhaps some clarifications.
I think the blog post you talk of is this one (included in the section that comes after the "repetition section): https://calendar.perfplanet.com/2010/bulletproof-javascript-benchmarks/