criterion performance measurements
overview
want to understand this report?
Grayscale/massiv
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.941637208936127e-3 | 3.98725426633969e-3 | 4.035428266915021e-3 |
| Standard deviation | 1.076306418346298e-4 | 1.448618638714054e-4 | 1.8274688593842667e-4 |
Outlying measurements have moderate (0.18947778360984616%) effect on estimated standard deviation.
Grayscale/repa
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.21365566692745e-3 | 5.241057565680461e-3 | 5.283478858480473e-3 |
| Standard deviation | 6.93575720198849e-5 | 1.020798961146495e-4 | 1.3966123853213875e-4 |
Outlying measurements have slight (2.4374999999999872e-2%) effect on estimated standard deviation.
Grayscale/accelerate
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.464724211072784e-3 | 1.5042537314593112e-3 | 1.5642888785084741e-3 |
| Standard deviation | 1.2555263951812768e-4 | 1.8289252441656614e-4 | 2.599507368722209e-4 |
Outlying measurements have severe (0.783082894200471%) effect on estimated standard deviation.
Blur/massiv
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.469320747123401e-3 | 7.794989764883741e-3 | 8.24904904287391e-3 |
| Standard deviation | 7.739707043359388e-4 | 1.1035556569393321e-3 | 1.471852121483367e-3 |
Outlying measurements have severe (0.7147775245744696%) effect on estimated standard deviation.
Blur/repa
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.623803904151717e-3 | 7.696344656097422e-3 | 7.789239783190264e-3 |
| Standard deviation | 1.8045889823584515e-4 | 2.3879922562424622e-4 | 3.482067508707071e-4 |
Outlying measurements have moderate (0.11104002629205367%) effect on estimated standard deviation.
Blur/accelerate
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.3304190597377385e-3 | 6.40781092276255e-3 | 6.526565221138957e-3 |
| Standard deviation | 2.2358868192641018e-4 | 2.934366081067699e-4 | 3.909241969725392e-4 |
Outlying measurements have moderate (0.23132833611193934%) effect on estimated standard deviation.
Gradient/massiv
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.219463448240674e-3 | 9.731095927201104e-3 | 1.0701961083169223e-2 |
| Standard deviation | 1.0238892801174083e-3 | 1.7911126498053095e-3 | 2.9245386850807764e-3 |
Outlying measurements have severe (0.7967756840321633%) effect on estimated standard deviation.
Gradient/repa
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.401943478575842e-2 | 1.4491869379992589e-2 | 1.5430727659438925e-2 |
| Standard deviation | 8.969153789125003e-4 | 1.7397220295739197e-3 | 2.807723347727379e-3 |
Outlying measurements have severe (0.5843875300920595%) effect on estimated standard deviation.
Gradient/accelerate
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.212494721300844e-3 | 9.386918760484433e-3 | 9.563631228081805e-3 |
| Standard deviation | 4.5974313153829456e-4 | 5.543710758073935e-4 | 6.879038650082912e-4 |
Outlying measurements have moderate (0.3075429963853639%) effect on estimated standard deviation.
Suppress/massiv
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.3884053866519185e-3 | 2.438752888336726e-3 | 2.4964632632917217e-3 |
| Standard deviation | 1.419362044065027e-4 | 1.7298040863950202e-4 | 2.1822961384441753e-4 |
Outlying measurements have moderate (0.4942263265231579%) effect on estimated standard deviation.
Suppress/repa
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.691769470589583e-3 | 2.7366146242370536e-3 | 2.8319710659452913e-3 |
| Standard deviation | 1.432115833511263e-4 | 2.1269452495329953e-4 | 3.408478943663942e-4 |
Outlying measurements have severe (0.5337319756334248%) effect on estimated standard deviation.
Suppress/accelerate
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.844797994451135e-3 | 4.9974216222776245e-3 | 5.203273557896451e-3 |
| Standard deviation | 3.4268601485679145e-4 | 4.905959837443568e-4 | 7.850921733059695e-4 |
Outlying measurements have severe (0.5950535538522186%) effect on estimated standard deviation.
Strong/massiv
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.740115079442367e-3 | 3.7519046572742835e-3 | 3.7657130425360814e-3 |
| Standard deviation | 3.453713859905818e-5 | 4.2244141559966464e-5 | 5.02517314564102e-5 |
Outlying measurements have slight (2.126654064272205e-2%) effect on estimated standard deviation.
Strong/repa
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.2225040153293069e-2 | 1.2811645428335107e-2 | 1.4504674235912838e-2 |
| Standard deviation | 1.1290252603879658e-3 | 2.1765682323666006e-3 | 3.9596655577890496e-3 |
Outlying measurements have severe (0.7656485661937449%) effect on estimated standard deviation.
Strong/accelerate
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.2129354432282182e-2 | 1.224493881293562e-2 | 1.2366453389505665e-2 |
| Standard deviation | 2.6347219860207923e-4 | 3.144408033045978e-4 | 4.2365115447748537e-4 |
Outlying measurements have slight (6.925876498377478e-2%) effect on estimated standard deviation.
Wildfire/massiv
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.7633391568039202e-3 | 1.8237434208927146e-3 | 2.023032465576844e-3 |
| Standard deviation | 4.6029627845741274e-5 | 3.486789445827805e-4 | 7.328749209377474e-4 |
Outlying measurements have severe (0.892458641189593%) effect on estimated standard deviation.
Wildfire/repa
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.0295869608529e-3 | 4.0946728170958594e-3 | 4.353206680233876e-3 |
| Standard deviation | 3.4358149865223325e-5 | 3.8748435875320553e-4 | 8.169622095261376e-4 |
Outlying measurements have severe (0.598704308570296%) effect on estimated standard deviation.
Wildfire/accelerate
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.289787417894477e-2 | 4.4206464654633175e-2 | 4.6901213573469776e-2 |
| Standard deviation | 7.512914201407016e-4 | 3.5144930994803355e-3 | 5.819986195120997e-3 |
Outlying measurements have moderate (0.27129970281154164%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.