Introduction
What is experiment module and when to use it.
What is experiment module and when to use it.
This module enables robust A/B testing following the Bayesian approach and to facilitate making binary decisions.
A typical use case for this is the following: an ML model is deployed for a subset of the population (of users/machines/cars, etc.), and we want to check how it affects the downstream metrics by comparing their values with the control group (the rest of the population that is not affected by the model's predictions). We usually want the answer as soon as possible to roll it out to the whole population if it has the desired effect on the metric of interest or take it down if not. An example could be a churn of streaming service subscribers after changing the video recommendation model.