From 5d08a2c31575f18ef5b74371575bcc6117542186 Mon Sep 17 00:00:00 2001 From: Michael Snoyman Date: Thu, 18 May 2017 09:41:47 -0700 Subject: [PATCH] Add reverse-bench.html --- static/reverse-bench.html | 1886 +++++++++++++++++++++++++++++++++++++ 1 file changed, 1886 insertions(+) create mode 100644 static/reverse-bench.html diff --git a/static/reverse-bench.html b/static/reverse-bench.html new file mode 100644 index 0000000..bd65054 --- /dev/null +++ b/static/reverse-bench.html @@ -0,0 +1,1886 @@ + + + + + criterion report + + + + + + + +
+
+

criterion performance measurements

+ +

overview

+ +

want to understand this report?

+ +
+ +

5/reverse

+ + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
lower boundestimateupper bound
OLS regressionxxxxxxxxx
R² goodness-of-fitxxxxxxxxx
Mean execution time2.1825676261906123e-82.2358490252398955e-82.37844799339728e-8
Standard deviation1.0410718832927102e-92.679739388766841e-95.028759887265124e-9
+ + +

Outlying measurements have severe + (0.9412781297226944%) + effect on estimated standard deviation.

+ +

5/myReverse

+ + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
lower boundestimateupper bound
OLS regressionxxxxxxxxx
R² goodness-of-fitxxxxxxxxx
Mean execution time4.3976666496170086e-84.4436941145299556e-84.485953844319826e-8
Standard deviation1.1846412312215601e-91.4129818567543152e-91.7437750926423863e-9
+ + +

Outlying measurements have severe + (0.5048047041113867%) + effect on estimated standard deviation.

+ +

5/betterReverse

+ + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
lower boundestimateupper bound
OLS regressionxxxxxxxxx
R² goodness-of-fitxxxxxxxxx
Mean execution time2.2468511683032136e-82.275122735723411e-82.308338369611327e-8
Standard deviation8.368738485812947e-101.0551141730817191e-91.4280336158690675e-9
+ + +

Outlying measurements have severe + (0.7000303924411976%) + effect on estimated standard deviation.

+ +

5/vectorReverse

+ + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
lower boundestimateupper bound
OLS regressionxxxxxxxxx
R² goodness-of-fitxxxxxxxxx
Mean execution time4.113268125528561e-74.1671732796036996e-74.2302707007282097e-7
Standard deviation1.497270976211413e-82.0058069721556883e-82.7911287812056826e-8
+ + +

Outlying measurements have severe + (0.6633637246704727%) + effect on estimated standard deviation.

+ +

5/svectorReverse

+ + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
lower boundestimateupper bound
OLS regressionxxxxxxxxx
R² goodness-of-fitxxxxxxxxx
Mean execution time6.920398407902705e-86.975841936879646e-87.046564991423694e-8
Standard deviation1.8066599476484786e-92.172896214537708e-92.6467403234113016e-9
+ + +

Outlying measurements have moderate + (0.48839555023650844%) + effect on estimated standard deviation.

+ +

5/uvectorReverse

+ + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
lower boundestimateupper bound
OLS regressionxxxxxxxxx
R² goodness-of-fitxxxxxxxxx
Mean execution time6.851679597092373e-86.925843908280533e-87.01740396232455e-8
Standard deviation2.210327892769043e-92.927763932650304e-94.326065221198047e-9
+ + +

Outlying measurements have severe + (0.6400701063900864%) + effect on estimated standard deviation.

+ +

100/reverse

+ + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
lower boundestimateupper bound
OLS regressionxxxxxxxxx
R² goodness-of-fitxxxxxxxxx
Mean execution time4.4247980578490166e-74.486931689301164e-74.715765238594911e-7
Standard deviation1.384237597896298e-83.4265713859801746e-87.91561949401906e-8
+ + +

Outlying measurements have severe + (0.8337723416690731%) + effect on estimated standard deviation.

+ +

100/myReverse

+ + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
lower boundestimateupper bound
OLS regressionxxxxxxxxx
R² goodness-of-fitxxxxxxxxx
Mean execution time1.007891519190093e-61.0195661875784653e-61.039020302638846e-6
Standard deviation3.817967123630337e-85.2243289403871843e-87.427452481277e-8
+ + +

Outlying measurements have severe + (0.6738845808731594%) + effect on estimated standard deviation.

+ +

100/betterReverse

+ + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
lower boundestimateupper bound
OLS regressionxxxxxxxxx
R² goodness-of-fitxxxxxxxxx
Mean execution time4.295364364477159e-74.3298605299216234e-74.3799217841438554e-7
Standard deviation1.112311803558022e-81.3612046474153825e-81.7582641093090672e-8
+ + +

Outlying measurements have moderate + (0.4540266643025085%) + effect on estimated standard deviation.

+ +

100/vectorReverse

+ + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
lower boundestimateupper bound
OLS regressionxxxxxxxxx
R² goodness-of-fitxxxxxxxxx
Mean execution time3.3057291168636383e-63.38274136179026e-63.548252976134029e-6
Standard deviation1.0608095029319091e-73.541439684399015e-76.096734294907109e-7
+ + +

Outlying measurements have severe + (0.8869680939351119%) + effect on estimated standard deviation.

+ +

100/svectorReverse

+ + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
lower boundestimateupper bound
OLS regressionxxxxxxxxx
R² goodness-of-fitxxxxxxxxx
Mean execution time6.866557693995546e-76.934741716236048e-77.021896276554826e-7
Standard deviation2.1479242834733937e-82.4882765696493914e-83.0244553441558444e-8
+ + +

Outlying measurements have severe + (0.5098333358544204%) + effect on estimated standard deviation.

+ +

100/uvectorReverse

+ + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
lower boundestimateupper bound
OLS regressionxxxxxxxxx
R² goodness-of-fitxxxxxxxxx
Mean execution time6.6949455396143e-76.763147099577562e-76.8492834272938e-7
Standard deviation2.090563167335844e-82.6756765803271828e-83.911103210894066e-8
+ + +

Outlying measurements have severe + (0.560253509340352%) + effect on estimated standard deviation.

+ +

10000/reverse

+ + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
lower boundestimateupper bound
OLS regressionxxxxxxxxx
R² goodness-of-fitxxxxxxxxx
Mean execution time8.226367938376666e-58.404068339648452e-58.709085964902236e-5
Standard deviation4.522568050534064e-67.86691139839666e-61.430260632818422e-5
+ + +

Outlying measurements have severe + (0.8002504670977215%) + effect on estimated standard deviation.

+ +

10000/myReverse

+ + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
lower boundestimateupper bound
OLS regressionxxxxxxxxx
R² goodness-of-fitxxxxxxxxx
Mean execution time9.89039039905634e-41.018843258186528e-31.0672437545241342e-3
Standard deviation8.176285380106038e-51.257903479559935e-42.101625919477303e-4
+ + +

Outlying measurements have severe + (0.8045762396300628%) + effect on estimated standard deviation.

+ +

10000/betterReverse

+ + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
lower boundestimateupper bound
OLS regressionxxxxxxxxx
R² goodness-of-fitxxxxxxxxx
Mean execution time8.037007935207724e-58.094225189166519e-58.184021433892617e-5
Standard deviation1.6701071654065847e-62.23963860459826e-63.3025989615865156e-6
+ + +

Outlying measurements have moderate + (0.25266616390534796%) + effect on estimated standard deviation.

+ +

10000/vectorReverse

+ + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
lower boundestimateupper bound
OLS regressionxxxxxxxxx
R² goodness-of-fitxxxxxxxxx
Mean execution time4.671121652397968e-44.874406423516324e-45.260025225863243e-4
Standard deviation5.3060569609557984e-58.807508602942347e-51.399993272699004e-4
+ + +

Outlying measurements have severe + (0.9129058424341375%) + effect on estimated standard deviation.

+ +

10000/svectorReverse

+ + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
lower boundestimateupper bound
OLS regressionxxxxxxxxx
R² goodness-of-fitxxxxxxxxx
Mean execution time8.363981457710693e-58.465469716167873e-58.572704740430688e-5
Standard deviation2.8557446230897197e-63.46327049505151e-64.740792412320742e-6
+ + +

Outlying measurements have moderate + (0.4309142876000203%) + effect on estimated standard deviation.

+ +

10000/uvectorReverse

+ + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
lower boundestimateupper bound
OLS regressionxxxxxxxxx
R² goodness-of-fitxxxxxxxxx
Mean execution time8.180549446549271e-58.302433383267748e-58.507035683076922e-5
Standard deviation3.0654232002257616e-65.168280070836902e-68.482840131547177e-6
+ + +

Outlying measurements have severe + (0.6386877066044123%) + effect on estimated standard deviation.

+ +

1000000/reverse

+ + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
lower boundestimateupper bound
OLS regressionxxxxxxxxx
R² goodness-of-fitxxxxxxxxx
Mean execution time5.82335887983178e-26.070360608160647e-26.351763956607893e-2
Standard deviation3.493123458566975e-34.869657126862904e-36.661851617464323e-3
+ + +

Outlying measurements have moderate + (0.23735406305053863%) + effect on estimated standard deviation.

+ +

1000000/myReverse

+ + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
lower boundestimateupper bound
OLS regressionxxxxxxxxx
R² goodness-of-fitxxxxxxxxx
Mean execution time0.15210531904438650.156169560005298550.16591606631843145
Standard deviation1.4696875602100575e-38.218916793373316e-31.2116156633981663e-2
+ + +

Outlying measurements have moderate + (0.12641003608244392%) + effect on estimated standard deviation.

+ +

1000000/betterReverse

+ + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
lower boundestimateupper bound
OLS regressionxxxxxxxxx
R² goodness-of-fitxxxxxxxxx
Mean execution time5.8424205701204296e-26.046765499354529e-26.389965309082378e-2
Standard deviation2.7554816954297187e-34.6048879250182835e-36.9992153869534645e-3
+ + +

Outlying measurements have moderate + (0.23596776268571917%) + effect on estimated standard deviation.

+ +

1000000/vectorReverse

+ + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
lower boundestimateupper bound
OLS regressionxxxxxxxxx
R² goodness-of-fitxxxxxxxxx
Mean execution time5.391284686073217e-25.609325219775799e-25.804273318320527e-2
Standard deviation2.5813403288497466e-33.581623161602982e-35.10770701971325e-3
+ + +

Outlying measurements have moderate + (0.15779515270543848%) + effect on estimated standard deviation.

+ +

1000000/svectorReverse

+ + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
lower boundestimateupper bound
OLS regressionxxxxxxxxx
R² goodness-of-fitxxxxxxxxx
Mean execution time1.3087657836357023e-21.3486536819947735e-21.4577670676972554e-2
Standard deviation6.632674890790757e-41.5583727257841432e-32.871073235275729e-3
+ + +

Outlying measurements have severe + (0.5619420802481624%) + effect on estimated standard deviation.

+ +

1000000/uvectorReverse

+ + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
lower boundestimateupper bound
OLS regressionxxxxxxxxx
R² goodness-of-fitxxxxxxxxx
Mean execution time1.207082513587815e-21.2224248519543568e-21.2365601516715435e-2
Standard deviation3.05961188737579e-43.8840682394674676e-44.906517806114332e-4
+ + +

Outlying measurements have moderate + (0.10403423030173367%) + effect on estimated standard deviation.

+ + +

understanding this report

+ +

In this report, each function benchmarked by criterion is assigned + a section of its own. The charts in each section are active; if + you hover your mouse over data points and annotations, you will see + more details.

+ +
    +
  • The chart on the left is a + kernel + density estimate (also known as a KDE) of time + measurements. This graphs the probability of any given time + measurement occurring. A spike indicates that a measurement of a + particular time occurred; its height indicates how often that + measurement was repeated.
    • + +
  • The chart on the right is the raw data from which the kernel + density estimate is built. The x axis indicates the + number of loop iterations, while the y axis shows measured + execution time for the given number of loop iterations. The + line behind the values is the linear regression prediction of + execution time for a given number of iterations. Ideally, all + measurements will be on (or very near) this line.
    • +
+ +

Under the charts is a small table. + The first two rows are the results of a linear regression run + on the measurements displayed in the right-hand chart.

+ +
    +
  • OLS regression indicates the + time estimated for a single loop iteration using an ordinary + least-squares regression model. This number is more accurate + than the mean estimate below it, as it more effectively + eliminates measurement overhead and other constant factors.
    • +
  • R² goodness-of-fit is a measure of how + accurately the linear regression model fits the observed + measurements. If the measurements are not too noisy, R² + should lie between 0.99 and 1, indicating an excellent fit. If + the number is below 0.99, something is confounding the accuracy + of the linear model.
    • +
  • Mean execution time and standard deviation are + statistics calculated from execution time + divided by number of iterations.
    • +
+ +

We use a statistical technique called + the bootstrap + to provide confidence intervals on our estimates. The + bootstrap-derived upper and lower bounds on estimates let you see + how accurate we believe those estimates to be. (Hover the mouse + over the table headers to see the confidence levels.)

+ +

A noisy benchmarking environment can cause some or many + measurements to fall far from the mean. These outlying + measurements can have a significant inflationary effect on the + estimate of the standard deviation. We calculate and display an + estimate of the extent to which the standard deviation has been + inflated by outliers.

+ + + +
+
+ + +