How to get data ready for NannyML

This tutorial is a practical guide on how to get your data in the right format to monitor your models with NannyML. We will illustrate step-by-step how to prepare data for monitoring. We will start with building a simple model that we will monitor later.

Step 1: Have a trained model (prerequisite)

If you are looking into monitoring, you probably already have a model, a training set, and most likely some production (monitored) data on which you would like to check how your model is doing. This tutorial uses the Banana Quality Prediction dataset to illustrate how to prepare your data.

Let's start by partitioning our data into three sets:

Training
Testing
Monitored (production data)

In a real scenario, you won't need to create the monitoring partition since this will naturally come after deploying the model.

If you already have a trained model, skip to Step 2: Building a reference dataset.

import pandas as pd

url = 'https://raw.githubusercontent.com/NannyML/sample_datasets/main/banana_quality/banana_quality.csv' 
data = pd.read_csv(url)
train_df, test_df, monitor_df = data.iloc[0:4000], data.iloc[4000:6000], data.iloc[6000:8000]

data.head()

Identifier

Timestamp

Size

Weight

Sweetness

Softness

HarvestTime

Ripeness

Acidity

Quality

The Quality column is the target variable. It is encoded as:

1 - Good banana quality
0 - Bad banana quality

Once we have the data separated, we can build a simple model.

from lightgbm import LGBMClassifier

features = [
    'Size', 'Weight', 'Sweetness', 'Softness', 
    'HarvestTime', 'Ripeness', 'Acidity'
]
target = 'Quality'
timestamp = 'Timestamp'

clf = LGBMClassifier()
clf.fit(train_df[features], train_df[target])

Let's quickly check its performance on the training and testing sets before moving to the next step.

from sklearn.metrics import roc_auc_score

roc_auc_train = roc_auc_score(
    train_df[target], clf.predict_proba(train_df[features])[:, 1]
)
roc_auc_test = roc_auc_score(
    test_df[target], clf.predict_proba(test_df[features])[:, 1]
)

print(f"ROC-AUC (train): {roc_auc_train}")
print(f"ROC-AUC (test): {roc_auc_test}")

ROC-AUC (train): 1.0
ROC-AUC (test): 0.9931109775641026

Looks like the model is doing a nice job in the training and testing datasets.

Step 2: Building a reference dataset

Once we have a model that performs well on the test set, we can use it to build a reference dataset.

Reference dataset: The reference dataset is a benchmark set where the model behaves as expected. It is important that this dataset was not seen by the model during training. It should include targets and the model's predictions. In this tutorial, we will use the test set to build the reference set.

NannyML uses the reference dataset to establish a baseline of how the monitoring data and the model predictions should look after deployment. Internally, it uses it to calibrate most of its performance estimation methods and calculate the default threshold values for every monitored metric.

To transform the test dataset into a reference dataset, all we need to do is add the predicted probabilities and the model predictions to the test dataset.

This means that we will create the following columns:

y_pred_proba: contains the model's predicted probabilities (scores) for class 1 (Good quality banana, in this case).
y_pred: contains the model's predicted class (0 or 1).

reference_df = test_df.copy()

reference_df['y_pred_proba'] = clf.predict_proba(test_df[features])[:, 1]
reference_df['y_pred'] = clf.predict(test_df[features])

reference_df.head()

We recommend storing your data as parquet files.

NannyML Cloud supports both parquet and CSV files, but CSV files don't store data type information. CSV files may cause incorrect data types to be inferred. If you later add more data to the model using the SDK or using parquet format, a data type conflict may occur.

Step 3: Setting up the monitored dataset

Monitored dataset: This is the data that NannyML will monitor. It typically contains the latest production data, which should be after the reference period ends. This dataset does not require targets.

Similarly, as with the reference dataset, let's add the following columns to the monitored dataset.

y_pred_proba: contains the model's predicted probabilities (scores) for class 1 (Good quality banana, in this case).
y_pred: contains the model's predicted class (0 or 1).

monitor_df['y_pred_proba'] = clf.predict_proba(monitor_df[features])[:, 1]
monitor_df['y_pred'] = clf.predict(monitor_df[features])

monitor_df.head()

In this particular example, we are dealing with a binary classification problem. The required columns for multiclass classification and regression differ slightly.

Here is a summary of all the columns required for regression, binary classification, and multiclass classification.

Regression Model - Required columns

Classification Model - Required Columns

If the model is multiclass classification, there should be a y_pred_proba column for each class.

Example:

y_pred_proba_setosa
y_pred_proba_virginica
y_pred_proba_versicolor

What if my data doesn't have timestamps?

Timestamps indicate the date and time when observations were recorded. If your dataset doesn't have timestamps, you can generate synthetic ones, just make sure they align with your specific use case.

For example:

import datetime as dt

timestamps = [dt.datetime(2020,1,1) + dt.timedelta(hours=x/2) for x in df.index]

Where to go next?

PreviousData Preparation NextTutorials