Monitoring a text classification model
Tutorial explaining how to monitor text classification models with NannyML
In this tutorial, we will use nannyML cloud to monitor a sentiment analysis text classification model where the model's goal is to predict the sentiment (Negative, Neutral, Positive) of a review left on Amazon.
The model and dataset
We will use a model trained on a subset of the Multilingual Amazon Reviews Dataset. The trained model can be found in the nannyML's hugging face hub.
For details of how this model was produced, check out the blog post: Are your NLP models deteriorating post-deployment? Let’s use unlabelled data to find out.
Reference and analysis sets
To evaluate the model in production, we have two sets:
Reference set - which contains all model inputs along with the model’s predictions and labels. This set establishes a baseline for every metric we want to monitor. Find the reference set: https://raw.githubusercontent.com/NannyML/sample_datasets/main/amazon_reviews/amazon_reviews_reference.csv
Analysis set - which contains all model inputs extracted from a production set with the model’s prediction, and in this case, labels. The analysis set is where NannyML analyzes/monitors the model’s performance and data drift of the model using the knowledge gained from the reference set. Find the analysis set: https://raw.githubusercontent.com/NannyML/sample_datasets/main/amazon_reviews/amazon_reviews_analysis_targets.csv
Monitoring with nannyML Cloud
Step 1: Add a new model
Click the Add model button to create a new model on your nannyML cloud dashboard.
Step 2: Define the problem type and main metric
Each review that we are classifying can be Negative, Positive, or Neutral. For this reason, we will set the problem type as Multiclass classification.
We will be monitoring the model's F1-score on a weekly basis.
Step 3: Configure the Reference set
Select "Upload via public link".
Use the following public URL to link the Reference dataset: https://raw.githubusercontent.com/NannyML/sample_datasets/main/amazon_reviews/amazon_reviews_reference.csv
Step 4: Define the reference dataset schema
Select the column timestamp as the Timestamp column
Select the column predicted_sentiment as the Prediction column
Select the real_sentiment as the Target column
Flag the columns negative_sentiment_pred_proba, neutral_sentiment_pred_proba, positive_sentiment_pred_proba as Prediction Scores.
Step 5: Configure the Analysis set
Select "Upload via public link".
Use the following public URL to link the Analysis dataset: https://raw.githubusercontent.com/NannyML/sample_datasets/main/amazon_reviews/amazon_reviews_analysis_targets.csv