Monitoring a text classification model
Tutorial explaining how to monitor text classification models with NannyML
Tutorial explaining how to monitor text classification models with NannyML
In this tutorial, we will use nannyML cloud to monitor a sentiment analysis text classification model where the model's goal is to predict the sentiment (Negative, Neutral, Positive) of a review left on Amazon.
We will use a model trained on a subset of the Multilingual Amazon Reviews Dataset. The trained model can be found in the nannyML's hugging face hub.
For details of how this model was produced, check out the blog post: Are your NLP models deteriorating post-deployment? Let’s use unlabelled data to find out.
To evaluate the model in production, we have two sets:
Reference set - which contains all model inputs along with the model’s predictions and labels. This set establishes a baseline for every metric we want to monitor. Find the reference set: https://raw.githubusercontent.com/NannyML/sample_datasets/main/amazon_reviews/amazon_reviews_reference.csv
Analysis set - which contains all model inputs extracted from a production set with the model’s prediction, and in this case, labels. The analysis set is where NannyML analyzes/monitors the model’s performance and data drift of the model using the knowledge gained from the reference set. Find the analysis set: https://raw.githubusercontent.com/NannyML/sample_datasets/main/amazon_reviews/amazon_reviews_analysis_targets.csv
Click the Add model button to create a new model on your nannyML cloud dashboard.
Each review that we are classifying can be Negative, Positive, or Neutral. For this reason, we will set the problem type as Multiclass classification.
We will be monitoring the model's F1-score on a weekly basis.
Select "Upload via public link".
Use the following public URL to link the Reference dataset: https://raw.githubusercontent.com/NannyML/sample_datasets/main/amazon_reviews/amazon_reviews_reference.csv
Select the column timestamp as the Timestamp column
Select the column predicted_sentiment as the Prediction column
Select the real_sentiment as the Target column
Flag the columns negative_sentiment_pred_proba, neutral_sentiment_pred_proba, positive_sentiment_pred_proba as Prediction Scores.
Select "Upload via public link".
Use the following public URL to link the Analysis dataset: https://raw.githubusercontent.com/NannyML/sample_datasets/main/amazon_reviews/amazon_reviews_analysis_targets.csv
