criterion performance measurements
overview
want to understand this report?
5/reverse
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.1825676261906123e-8 | 2.2358490252398955e-8 | 2.37844799339728e-8 |
| Standard deviation | 1.0410718832927102e-9 | 2.679739388766841e-9 | 5.028759887265124e-9 |
Outlying measurements have severe (0.9412781297226944%) effect on estimated standard deviation.
5/myReverse
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.3976666496170086e-8 | 4.4436941145299556e-8 | 4.485953844319826e-8 |
| Standard deviation | 1.1846412312215601e-9 | 1.4129818567543152e-9 | 1.7437750926423863e-9 |
Outlying measurements have severe (0.5048047041113867%) effect on estimated standard deviation.
5/betterReverse
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.2468511683032136e-8 | 2.275122735723411e-8 | 2.308338369611327e-8 |
| Standard deviation | 8.368738485812947e-10 | 1.0551141730817191e-9 | 1.4280336158690675e-9 |
Outlying measurements have severe (0.7000303924411976%) effect on estimated standard deviation.
5/vectorReverse
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.113268125528561e-7 | 4.1671732796036996e-7 | 4.2302707007282097e-7 |
| Standard deviation | 1.497270976211413e-8 | 2.0058069721556883e-8 | 2.7911287812056826e-8 |
Outlying measurements have severe (0.6633637246704727%) effect on estimated standard deviation.
5/svectorReverse
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.920398407902705e-8 | 6.975841936879646e-8 | 7.046564991423694e-8 |
| Standard deviation | 1.8066599476484786e-9 | 2.172896214537708e-9 | 2.6467403234113016e-9 |
Outlying measurements have moderate (0.48839555023650844%) effect on estimated standard deviation.
5/uvectorReverse
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.851679597092373e-8 | 6.925843908280533e-8 | 7.01740396232455e-8 |
| Standard deviation | 2.210327892769043e-9 | 2.927763932650304e-9 | 4.326065221198047e-9 |
Outlying measurements have severe (0.6400701063900864%) effect on estimated standard deviation.
100/reverse
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.4247980578490166e-7 | 4.486931689301164e-7 | 4.715765238594911e-7 |
| Standard deviation | 1.384237597896298e-8 | 3.4265713859801746e-8 | 7.91561949401906e-8 |
Outlying measurements have severe (0.8337723416690731%) effect on estimated standard deviation.
100/myReverse
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.007891519190093e-6 | 1.0195661875784653e-6 | 1.039020302638846e-6 |
| Standard deviation | 3.817967123630337e-8 | 5.2243289403871843e-8 | 7.427452481277e-8 |
Outlying measurements have severe (0.6738845808731594%) effect on estimated standard deviation.
100/betterReverse
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.295364364477159e-7 | 4.3298605299216234e-7 | 4.3799217841438554e-7 |
| Standard deviation | 1.112311803558022e-8 | 1.3612046474153825e-8 | 1.7582641093090672e-8 |
Outlying measurements have moderate (0.4540266643025085%) effect on estimated standard deviation.
100/vectorReverse
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.3057291168636383e-6 | 3.38274136179026e-6 | 3.548252976134029e-6 |
| Standard deviation | 1.0608095029319091e-7 | 3.541439684399015e-7 | 6.096734294907109e-7 |
Outlying measurements have severe (0.8869680939351119%) effect on estimated standard deviation.
100/svectorReverse
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.866557693995546e-7 | 6.934741716236048e-7 | 7.021896276554826e-7 |
| Standard deviation | 2.1479242834733937e-8 | 2.4882765696493914e-8 | 3.0244553441558444e-8 |
Outlying measurements have severe (0.5098333358544204%) effect on estimated standard deviation.
100/uvectorReverse
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.6949455396143e-7 | 6.763147099577562e-7 | 6.8492834272938e-7 |
| Standard deviation | 2.090563167335844e-8 | 2.6756765803271828e-8 | 3.911103210894066e-8 |
Outlying measurements have severe (0.560253509340352%) effect on estimated standard deviation.
10000/reverse
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.226367938376666e-5 | 8.404068339648452e-5 | 8.709085964902236e-5 |
| Standard deviation | 4.522568050534064e-6 | 7.86691139839666e-6 | 1.430260632818422e-5 |
Outlying measurements have severe (0.8002504670977215%) effect on estimated standard deviation.
10000/myReverse
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.89039039905634e-4 | 1.018843258186528e-3 | 1.0672437545241342e-3 |
| Standard deviation | 8.176285380106038e-5 | 1.257903479559935e-4 | 2.101625919477303e-4 |
Outlying measurements have severe (0.8045762396300628%) effect on estimated standard deviation.
10000/betterReverse
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.037007935207724e-5 | 8.094225189166519e-5 | 8.184021433892617e-5 |
| Standard deviation | 1.6701071654065847e-6 | 2.23963860459826e-6 | 3.3025989615865156e-6 |
Outlying measurements have moderate (0.25266616390534796%) effect on estimated standard deviation.
10000/vectorReverse
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.671121652397968e-4 | 4.874406423516324e-4 | 5.260025225863243e-4 |
| Standard deviation | 5.3060569609557984e-5 | 8.807508602942347e-5 | 1.399993272699004e-4 |
Outlying measurements have severe (0.9129058424341375%) effect on estimated standard deviation.
10000/svectorReverse
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.363981457710693e-5 | 8.465469716167873e-5 | 8.572704740430688e-5 |
| Standard deviation | 2.8557446230897197e-6 | 3.46327049505151e-6 | 4.740792412320742e-6 |
Outlying measurements have moderate (0.4309142876000203%) effect on estimated standard deviation.
10000/uvectorReverse
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.180549446549271e-5 | 8.302433383267748e-5 | 8.507035683076922e-5 |
| Standard deviation | 3.0654232002257616e-6 | 5.168280070836902e-6 | 8.482840131547177e-6 |
Outlying measurements have severe (0.6386877066044123%) effect on estimated standard deviation.
1000000/reverse
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.82335887983178e-2 | 6.070360608160647e-2 | 6.351763956607893e-2 |
| Standard deviation | 3.493123458566975e-3 | 4.869657126862904e-3 | 6.661851617464323e-3 |
Outlying measurements have moderate (0.23735406305053863%) effect on estimated standard deviation.
1000000/myReverse
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.1521053190443865 | 0.15616956000529855 | 0.16591606631843145 |
| Standard deviation | 1.4696875602100575e-3 | 8.218916793373316e-3 | 1.2116156633981663e-2 |
Outlying measurements have moderate (0.12641003608244392%) effect on estimated standard deviation.
1000000/betterReverse
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.8424205701204296e-2 | 6.046765499354529e-2 | 6.389965309082378e-2 |
| Standard deviation | 2.7554816954297187e-3 | 4.6048879250182835e-3 | 6.9992153869534645e-3 |
Outlying measurements have moderate (0.23596776268571917%) effect on estimated standard deviation.
1000000/vectorReverse
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.391284686073217e-2 | 5.609325219775799e-2 | 5.804273318320527e-2 |
| Standard deviation | 2.5813403288497466e-3 | 3.581623161602982e-3 | 5.10770701971325e-3 |
Outlying measurements have moderate (0.15779515270543848%) effect on estimated standard deviation.
1000000/svectorReverse
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.3087657836357023e-2 | 1.3486536819947735e-2 | 1.4577670676972554e-2 |
| Standard deviation | 6.632674890790757e-4 | 1.5583727257841432e-3 | 2.871073235275729e-3 |
Outlying measurements have severe (0.5619420802481624%) effect on estimated standard deviation.
1000000/uvectorReverse
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.207082513587815e-2 | 1.2224248519543568e-2 | 1.2365601516715435e-2 |
| Standard deviation | 3.05961188737579e-4 | 3.8840682394674676e-4 | 4.906517806114332e-4 |
Outlying measurements have moderate (0.10403423030173367%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.