# Defaults for ROPE and estimation precision

## When are ROPE and Estimation Precision automatically calculated?

ROPE and estimation precision are typically business-related. ROPE is defined as a region with a similar business impact to the expected impact. Precision should be determined so that it limits the probability of getting a false positive (falsely accepting the hypothesis, that is - claiming the population performance is within ROPE) or false negative probability down to the business-required level. It is sometimes difficult to provide those parameters, yet it is still worth running a Probabilistic Model Evaluation. For those cases, NannyML provides reasonable defaults for ROPE and estimation precision.

## Default ROPE

Default ROPE reflects the hypothesis that an ML model performance *is no worse than the one from reference data*. In practice, default ROPE spans from the left 95% HDI edge from reference performance posterior to the maximum value of the metric possible (1 for the metrics currently supported). Figure 1 shows the default ROPE.

<figure><img src="https://content.gitbook.com/content/vipr4qR9MrP243sDPhAQ/blobs/248W5fSDHxuzo1HR4YHq/default_ROPE.png" alt=""><figcaption><p>Figure 1. Default ROPE based on reference data performance metric posterior.</p></figcaption></figure>

## <mark style="background-color:orange;">Default precision</mark>

The default precision is calculated to ensure the experiment's power is 0.8, with the experiment's goal to get a conclusive answer (to accept or reject the hypothesis). At the default precision, the experiment will yield a conclusive answer with 80% probability. The process of estimating the default precision is the following:

1. Assume that the hypothesis is correct: the performance metric is within ROPE.&#x20;
2. Sample performance metric uniformly from ROPE.
3. Generate *n* observations of data that we could observe given the performance metric sampled (similarly to predictive posterior sampling).&#x20;
4. Get posterior from the sampled observations.
5. Calculate the HDI of the posterior from 4.
6. Check whether HDI is fully within the ROPE.
7. Repeat steps 2-6 multiple times. Store the result from step 6.
8. Check if \~80% of the experiment results (step 6) give a conclusive answer.
9. Repeat steps 2-8 to find n, for which 80% of experiments give a conclusive, positive answer.
10. Repeat the process with the assumption that the hypothesis should be rejected (that is - sample performance metric from outside of ROPE) to find *n* for this assumption. Pick the larger *n*.