Introduction

What is experiment module and when to use it.

Experiments module

This module enables robust A/B testing following the Bayesian approach and HDI+ROPE decision rule to facilitate making binary decisions.

When to use it

A typical use case for this is the following: an ML model is deployed for a subset of the population (of users/machines/cars, etc.), and we want to check how it affects the downstream metrics by comparing their values with the control group (the rest of the population that is not affected by the model's predictions). We usually want the answer as soon as possible to roll it out to the whole population if it has the desired effect on the metric of interest or take it down if not. An example could be a churn of streaming service subscribers after changing the video recommendation model.