Data Preparation

Preparing your model data for NannyML

What data does NannyML need in order to perform model evaluation? We have a detailed tutorial regarding evaluating a binary classification model with NannyML. NannyML's model evaluation module only assesses whether the model's performance when deployed meets expectations with as little data as possible given a required statistical confidence. In order to do more comprehensive model monitoring over time then NannyML's model monitoring module should be used. Conceptually NannyML needs the following information:

• The model's predicted probabilities.

• The actual target values for the model's prediction during the reference period.

• The classification thresholds for the model's predicted probabilities.

How should that information be encoded for NannyML to consume it? The classification threshold is just a number that is provided during the add new evaluation model wizard. The predicted probabilities and targets are presented in two columns with each row representing a model prediction. Let's see an example:

predicted_probability	target
0.32	0
0.62	1
0.83	1

However, when providing additional evaluation data, the target column is optional. NannyML's model evaluation algorithm can make use of pure model predictions in order to increase its confidence in the evaluated model's performance.