+
criterion performance measurements
+
+
overview
+
+
want to understand this report?
+
+
+
+
+
+
+
+
+ |
+ lower bound |
+ estimate |
+ upper bound |
+
+
+
+ OLS regression |
+ xxx |
+ xxx |
+ xxx |
+
+
+ R² goodness-of-fit |
+ xxx |
+ xxx |
+ xxx |
+
+
+ Mean execution time |
+ 2.1825676261906123e-8 |
+ 2.2358490252398955e-8 |
+ 2.37844799339728e-8 |
+
+
+ Standard deviation |
+ 1.0410718832927102e-9 |
+ 2.679739388766841e-9 |
+ 5.028759887265124e-9 |
+
+
+
+
+
+ Outlying measurements have severe
+ (0.9412781297226944%)
+ effect on estimated standard deviation.
+
+
+
+
+
+
+ |
+ lower bound |
+ estimate |
+ upper bound |
+
+
+
+ OLS regression |
+ xxx |
+ xxx |
+ xxx |
+
+
+ R² goodness-of-fit |
+ xxx |
+ xxx |
+ xxx |
+
+
+ Mean execution time |
+ 4.3976666496170086e-8 |
+ 4.4436941145299556e-8 |
+ 4.485953844319826e-8 |
+
+
+ Standard deviation |
+ 1.1846412312215601e-9 |
+ 1.4129818567543152e-9 |
+ 1.7437750926423863e-9 |
+
+
+
+
+
+ Outlying measurements have severe
+ (0.5048047041113867%)
+ effect on estimated standard deviation.
+
+
+
+
+
+
+ |
+ lower bound |
+ estimate |
+ upper bound |
+
+
+
+ OLS regression |
+ xxx |
+ xxx |
+ xxx |
+
+
+ R² goodness-of-fit |
+ xxx |
+ xxx |
+ xxx |
+
+
+ Mean execution time |
+ 2.2468511683032136e-8 |
+ 2.275122735723411e-8 |
+ 2.308338369611327e-8 |
+
+
+ Standard deviation |
+ 8.368738485812947e-10 |
+ 1.0551141730817191e-9 |
+ 1.4280336158690675e-9 |
+
+
+
+
+
+ Outlying measurements have severe
+ (0.7000303924411976%)
+ effect on estimated standard deviation.
+
+
+
+
+
+
+ |
+ lower bound |
+ estimate |
+ upper bound |
+
+
+
+ OLS regression |
+ xxx |
+ xxx |
+ xxx |
+
+
+ R² goodness-of-fit |
+ xxx |
+ xxx |
+ xxx |
+
+
+ Mean execution time |
+ 4.113268125528561e-7 |
+ 4.1671732796036996e-7 |
+ 4.2302707007282097e-7 |
+
+
+ Standard deviation |
+ 1.497270976211413e-8 |
+ 2.0058069721556883e-8 |
+ 2.7911287812056826e-8 |
+
+
+
+
+
+ Outlying measurements have severe
+ (0.6633637246704727%)
+ effect on estimated standard deviation.
+
+
+
+
+
+
+ |
+ lower bound |
+ estimate |
+ upper bound |
+
+
+
+ OLS regression |
+ xxx |
+ xxx |
+ xxx |
+
+
+ R² goodness-of-fit |
+ xxx |
+ xxx |
+ xxx |
+
+
+ Mean execution time |
+ 6.920398407902705e-8 |
+ 6.975841936879646e-8 |
+ 7.046564991423694e-8 |
+
+
+ Standard deviation |
+ 1.8066599476484786e-9 |
+ 2.172896214537708e-9 |
+ 2.6467403234113016e-9 |
+
+
+
+
+
+ Outlying measurements have moderate
+ (0.48839555023650844%)
+ effect on estimated standard deviation.
+
+
+
+
+
+
+ |
+ lower bound |
+ estimate |
+ upper bound |
+
+
+
+ OLS regression |
+ xxx |
+ xxx |
+ xxx |
+
+
+ R² goodness-of-fit |
+ xxx |
+ xxx |
+ xxx |
+
+
+ Mean execution time |
+ 6.851679597092373e-8 |
+ 6.925843908280533e-8 |
+ 7.01740396232455e-8 |
+
+
+ Standard deviation |
+ 2.210327892769043e-9 |
+ 2.927763932650304e-9 |
+ 4.326065221198047e-9 |
+
+
+
+
+
+ Outlying measurements have severe
+ (0.6400701063900864%)
+ effect on estimated standard deviation.
+
+
+
+
+
+
+ |
+ lower bound |
+ estimate |
+ upper bound |
+
+
+
+ OLS regression |
+ xxx |
+ xxx |
+ xxx |
+
+
+ R² goodness-of-fit |
+ xxx |
+ xxx |
+ xxx |
+
+
+ Mean execution time |
+ 4.4247980578490166e-7 |
+ 4.486931689301164e-7 |
+ 4.715765238594911e-7 |
+
+
+ Standard deviation |
+ 1.384237597896298e-8 |
+ 3.4265713859801746e-8 |
+ 7.91561949401906e-8 |
+
+
+
+
+
+ Outlying measurements have severe
+ (0.8337723416690731%)
+ effect on estimated standard deviation.
+
+
+
+
+
+
+ |
+ lower bound |
+ estimate |
+ upper bound |
+
+
+
+ OLS regression |
+ xxx |
+ xxx |
+ xxx |
+
+
+ R² goodness-of-fit |
+ xxx |
+ xxx |
+ xxx |
+
+
+ Mean execution time |
+ 1.007891519190093e-6 |
+ 1.0195661875784653e-6 |
+ 1.039020302638846e-6 |
+
+
+ Standard deviation |
+ 3.817967123630337e-8 |
+ 5.2243289403871843e-8 |
+ 7.427452481277e-8 |
+
+
+
+
+
+ Outlying measurements have severe
+ (0.6738845808731594%)
+ effect on estimated standard deviation.
+
+
+
+
+
+
+ |
+ lower bound |
+ estimate |
+ upper bound |
+
+
+
+ OLS regression |
+ xxx |
+ xxx |
+ xxx |
+
+
+ R² goodness-of-fit |
+ xxx |
+ xxx |
+ xxx |
+
+
+ Mean execution time |
+ 4.295364364477159e-7 |
+ 4.3298605299216234e-7 |
+ 4.3799217841438554e-7 |
+
+
+ Standard deviation |
+ 1.112311803558022e-8 |
+ 1.3612046474153825e-8 |
+ 1.7582641093090672e-8 |
+
+
+
+
+
+ Outlying measurements have moderate
+ (0.4540266643025085%)
+ effect on estimated standard deviation.
+
+
+
+
+
+
+ |
+ lower bound |
+ estimate |
+ upper bound |
+
+
+
+ OLS regression |
+ xxx |
+ xxx |
+ xxx |
+
+
+ R² goodness-of-fit |
+ xxx |
+ xxx |
+ xxx |
+
+
+ Mean execution time |
+ 3.3057291168636383e-6 |
+ 3.38274136179026e-6 |
+ 3.548252976134029e-6 |
+
+
+ Standard deviation |
+ 1.0608095029319091e-7 |
+ 3.541439684399015e-7 |
+ 6.096734294907109e-7 |
+
+
+
+
+
+ Outlying measurements have severe
+ (0.8869680939351119%)
+ effect on estimated standard deviation.
+
+
+
+
+
+
+ |
+ lower bound |
+ estimate |
+ upper bound |
+
+
+
+ OLS regression |
+ xxx |
+ xxx |
+ xxx |
+
+
+ R² goodness-of-fit |
+ xxx |
+ xxx |
+ xxx |
+
+
+ Mean execution time |
+ 6.866557693995546e-7 |
+ 6.934741716236048e-7 |
+ 7.021896276554826e-7 |
+
+
+ Standard deviation |
+ 2.1479242834733937e-8 |
+ 2.4882765696493914e-8 |
+ 3.0244553441558444e-8 |
+
+
+
+
+
+ Outlying measurements have severe
+ (0.5098333358544204%)
+ effect on estimated standard deviation.
+
+
+
+
+
+
+ |
+ lower bound |
+ estimate |
+ upper bound |
+
+
+
+ OLS regression |
+ xxx |
+ xxx |
+ xxx |
+
+
+ R² goodness-of-fit |
+ xxx |
+ xxx |
+ xxx |
+
+
+ Mean execution time |
+ 6.6949455396143e-7 |
+ 6.763147099577562e-7 |
+ 6.8492834272938e-7 |
+
+
+ Standard deviation |
+ 2.090563167335844e-8 |
+ 2.6756765803271828e-8 |
+ 3.911103210894066e-8 |
+
+
+
+
+
+ Outlying measurements have severe
+ (0.560253509340352%)
+ effect on estimated standard deviation.
+
+
+
+
+
+
+ |
+ lower bound |
+ estimate |
+ upper bound |
+
+
+
+ OLS regression |
+ xxx |
+ xxx |
+ xxx |
+
+
+ R² goodness-of-fit |
+ xxx |
+ xxx |
+ xxx |
+
+
+ Mean execution time |
+ 8.226367938376666e-5 |
+ 8.404068339648452e-5 |
+ 8.709085964902236e-5 |
+
+
+ Standard deviation |
+ 4.522568050534064e-6 |
+ 7.86691139839666e-6 |
+ 1.430260632818422e-5 |
+
+
+
+
+
+ Outlying measurements have severe
+ (0.8002504670977215%)
+ effect on estimated standard deviation.
+
+
+
+
+
+
+ |
+ lower bound |
+ estimate |
+ upper bound |
+
+
+
+ OLS regression |
+ xxx |
+ xxx |
+ xxx |
+
+
+ R² goodness-of-fit |
+ xxx |
+ xxx |
+ xxx |
+
+
+ Mean execution time |
+ 9.89039039905634e-4 |
+ 1.018843258186528e-3 |
+ 1.0672437545241342e-3 |
+
+
+ Standard deviation |
+ 8.176285380106038e-5 |
+ 1.257903479559935e-4 |
+ 2.101625919477303e-4 |
+
+
+
+
+
+ Outlying measurements have severe
+ (0.8045762396300628%)
+ effect on estimated standard deviation.
+
+
+
+
+
+
+ |
+ lower bound |
+ estimate |
+ upper bound |
+
+
+
+ OLS regression |
+ xxx |
+ xxx |
+ xxx |
+
+
+ R² goodness-of-fit |
+ xxx |
+ xxx |
+ xxx |
+
+
+ Mean execution time |
+ 8.037007935207724e-5 |
+ 8.094225189166519e-5 |
+ 8.184021433892617e-5 |
+
+
+ Standard deviation |
+ 1.6701071654065847e-6 |
+ 2.23963860459826e-6 |
+ 3.3025989615865156e-6 |
+
+
+
+
+
+ Outlying measurements have moderate
+ (0.25266616390534796%)
+ effect on estimated standard deviation.
+
+
+
+
+
+
+ |
+ lower bound |
+ estimate |
+ upper bound |
+
+
+
+ OLS regression |
+ xxx |
+ xxx |
+ xxx |
+
+
+ R² goodness-of-fit |
+ xxx |
+ xxx |
+ xxx |
+
+
+ Mean execution time |
+ 4.671121652397968e-4 |
+ 4.874406423516324e-4 |
+ 5.260025225863243e-4 |
+
+
+ Standard deviation |
+ 5.3060569609557984e-5 |
+ 8.807508602942347e-5 |
+ 1.399993272699004e-4 |
+
+
+
+
+
+ Outlying measurements have severe
+ (0.9129058424341375%)
+ effect on estimated standard deviation.
+
+
+
+
+
+
+ |
+ lower bound |
+ estimate |
+ upper bound |
+
+
+
+ OLS regression |
+ xxx |
+ xxx |
+ xxx |
+
+
+ R² goodness-of-fit |
+ xxx |
+ xxx |
+ xxx |
+
+
+ Mean execution time |
+ 8.363981457710693e-5 |
+ 8.465469716167873e-5 |
+ 8.572704740430688e-5 |
+
+
+ Standard deviation |
+ 2.8557446230897197e-6 |
+ 3.46327049505151e-6 |
+ 4.740792412320742e-6 |
+
+
+
+
+
+ Outlying measurements have moderate
+ (0.4309142876000203%)
+ effect on estimated standard deviation.
+
+
+
+
+
+
+ |
+ lower bound |
+ estimate |
+ upper bound |
+
+
+
+ OLS regression |
+ xxx |
+ xxx |
+ xxx |
+
+
+ R² goodness-of-fit |
+ xxx |
+ xxx |
+ xxx |
+
+
+ Mean execution time |
+ 8.180549446549271e-5 |
+ 8.302433383267748e-5 |
+ 8.507035683076922e-5 |
+
+
+ Standard deviation |
+ 3.0654232002257616e-6 |
+ 5.168280070836902e-6 |
+ 8.482840131547177e-6 |
+
+
+
+
+
+ Outlying measurements have severe
+ (0.6386877066044123%)
+ effect on estimated standard deviation.
+
+
+
+
+
+
+ |
+ lower bound |
+ estimate |
+ upper bound |
+
+
+
+ OLS regression |
+ xxx |
+ xxx |
+ xxx |
+
+
+ R² goodness-of-fit |
+ xxx |
+ xxx |
+ xxx |
+
+
+ Mean execution time |
+ 5.82335887983178e-2 |
+ 6.070360608160647e-2 |
+ 6.351763956607893e-2 |
+
+
+ Standard deviation |
+ 3.493123458566975e-3 |
+ 4.869657126862904e-3 |
+ 6.661851617464323e-3 |
+
+
+
+
+
+ Outlying measurements have moderate
+ (0.23735406305053863%)
+ effect on estimated standard deviation.
+
+
+
+
+
+
+ |
+ lower bound |
+ estimate |
+ upper bound |
+
+
+
+ OLS regression |
+ xxx |
+ xxx |
+ xxx |
+
+
+ R² goodness-of-fit |
+ xxx |
+ xxx |
+ xxx |
+
+
+ Mean execution time |
+ 0.1521053190443865 |
+ 0.15616956000529855 |
+ 0.16591606631843145 |
+
+
+ Standard deviation |
+ 1.4696875602100575e-3 |
+ 8.218916793373316e-3 |
+ 1.2116156633981663e-2 |
+
+
+
+
+
+ Outlying measurements have moderate
+ (0.12641003608244392%)
+ effect on estimated standard deviation.
+
+
+
+
+
+
+ |
+ lower bound |
+ estimate |
+ upper bound |
+
+
+
+ OLS regression |
+ xxx |
+ xxx |
+ xxx |
+
+
+ R² goodness-of-fit |
+ xxx |
+ xxx |
+ xxx |
+
+
+ Mean execution time |
+ 5.8424205701204296e-2 |
+ 6.046765499354529e-2 |
+ 6.389965309082378e-2 |
+
+
+ Standard deviation |
+ 2.7554816954297187e-3 |
+ 4.6048879250182835e-3 |
+ 6.9992153869534645e-3 |
+
+
+
+
+
+ Outlying measurements have moderate
+ (0.23596776268571917%)
+ effect on estimated standard deviation.
+
+
+
+
+
+
+ |
+ lower bound |
+ estimate |
+ upper bound |
+
+
+
+ OLS regression |
+ xxx |
+ xxx |
+ xxx |
+
+
+ R² goodness-of-fit |
+ xxx |
+ xxx |
+ xxx |
+
+
+ Mean execution time |
+ 5.391284686073217e-2 |
+ 5.609325219775799e-2 |
+ 5.804273318320527e-2 |
+
+
+ Standard deviation |
+ 2.5813403288497466e-3 |
+ 3.581623161602982e-3 |
+ 5.10770701971325e-3 |
+
+
+
+
+
+ Outlying measurements have moderate
+ (0.15779515270543848%)
+ effect on estimated standard deviation.
+
+
+
+
+
+
+ |
+ lower bound |
+ estimate |
+ upper bound |
+
+
+
+ OLS regression |
+ xxx |
+ xxx |
+ xxx |
+
+
+ R² goodness-of-fit |
+ xxx |
+ xxx |
+ xxx |
+
+
+ Mean execution time |
+ 1.3087657836357023e-2 |
+ 1.3486536819947735e-2 |
+ 1.4577670676972554e-2 |
+
+
+ Standard deviation |
+ 6.632674890790757e-4 |
+ 1.5583727257841432e-3 |
+ 2.871073235275729e-3 |
+
+
+
+
+
+ Outlying measurements have severe
+ (0.5619420802481624%)
+ effect on estimated standard deviation.
+
+
+
+
+
+
+ |
+ lower bound |
+ estimate |
+ upper bound |
+
+
+
+ OLS regression |
+ xxx |
+ xxx |
+ xxx |
+
+
+ R² goodness-of-fit |
+ xxx |
+ xxx |
+ xxx |
+
+
+ Mean execution time |
+ 1.207082513587815e-2 |
+ 1.2224248519543568e-2 |
+ 1.2365601516715435e-2 |
+
+
+ Standard deviation |
+ 3.05961188737579e-4 |
+ 3.8840682394674676e-4 |
+ 4.906517806114332e-4 |
+
+
+
+
+
+ Outlying measurements have moderate
+ (0.10403423030173367%)
+ effect on estimated standard deviation.
+
+
+
+
+
In this report, each function benchmarked by criterion is assigned
+ a section of its own. The charts in each section are active; if
+ you hover your mouse over data points and annotations, you will see
+ more details.
+
+
+ - The chart on the left is a
+ kernel
+ density estimate (also known as a KDE) of time
+ measurements. This graphs the probability of any given time
+ measurement occurring. A spike indicates that a measurement of a
+ particular time occurred; its height indicates how often that
+ measurement was repeated.
+
+ - The chart on the right is the raw data from which the kernel
+ density estimate is built. The x axis indicates the
+ number of loop iterations, while the y axis shows measured
+ execution time for the given number of loop iterations. The
+ line behind the values is the linear regression prediction of
+ execution time for a given number of iterations. Ideally, all
+ measurements will be on (or very near) this line.
+
+
+
Under the charts is a small table.
+ The first two rows are the results of a linear regression run
+ on the measurements displayed in the right-hand chart.
+
+
+ - OLS regression indicates the
+ time estimated for a single loop iteration using an ordinary
+ least-squares regression model. This number is more accurate
+ than the mean estimate below it, as it more effectively
+ eliminates measurement overhead and other constant factors.
+ - R² goodness-of-fit is a measure of how
+ accurately the linear regression model fits the observed
+ measurements. If the measurements are not too noisy, R²
+ should lie between 0.99 and 1, indicating an excellent fit. If
+ the number is below 0.99, something is confounding the accuracy
+ of the linear model.
+ - Mean execution time and standard deviation are
+ statistics calculated from execution time
+ divided by number of iterations.
+
+
+
We use a statistical technique called
+ the bootstrap
+ to provide confidence intervals on our estimates. The
+ bootstrap-derived upper and lower bounds on estimates let you see
+ how accurate we believe those estimates to be. (Hover the mouse
+ over the table headers to see the confidence levels.)
+
+
A noisy benchmarking environment can cause some or many
+ measurements to fall far from the mean. These outlying
+ measurements can have a significant inflationary effect on the
+ estimate of the standard deviation. We calculate and display an
+ estimate of the extent to which the standard deviation has been
+ inflated by outliers.
+
+
+
+