NannyML Cloud
HomeBlogNannyML OSS Docs
v0.22.0
v0.22.0
  • ☂️Introduction
  • Model Monitoring
    • Quickstart
    • Data Preparation
      • How to get data ready for NannyML
    • Tutorials
      • Monitoring a tabular data model
      • Monitoring with segmentation
      • Monitoring a text classification model
      • Monitoring a computer vision model
    • How it works
      • Probabilistic Adaptive Performance Estimation (PAPE)
      • Reverse Concept Drift (RCD)
  • Product tour
    • Navigation
    • Adding a model
    • Model overview
    • Deleting a model
    • Model side panel
      • Summary
      • Performance
      • Concept drift
      • Covariate shift
      • Data quality
      • Logs
      • Model settings
        • General
        • Data
        • Performance settings
        • Concept Drift settings
        • Covariate Shift settings
        • Descriptive Statistics settings
        • Data Quality settings
    • Account settings
  • Deployment
    • Azure
      • Azure Managed Application
        • Finding the URL to access managed NannyML Cloud
        • Enabling access to storage
      • Azure Software-as-a-Service (SaaS)
    • AWS
      • AMI with CFT
        • Architecture
      • EKS
        • Quick start cluster setup
      • S3 Access
    • Application setup
      • Authentication
      • Notifications
      • Webhooks
      • Permissions
  • NannyML Cloud SDK
    • Getting Started
    • API Reference
  • Probabilistic Model Evaluation
    • Introduction
    • Tutorials
      • Evaluating a binary classification model
      • Data Preparation
    • How it works
      • HDI+ROPE (with minimum precision)
      • Getting Probability Distribution of a Performance Metric with targets
      • Getting Probability Distribution of Performance Metric without targets
      • Getting Probability Distribution of Performance Metric when some observations have labels
      • Defaults for ROPE and estimation precision
  • Experiments Module
    • Introduction
    • Tutorials
      • Running an A/B test
      • Data Preparation
    • How it works
      • Getting probability distribution of the difference of binary downstream metrics
  • miscellaneous
    • Engineering
    • Usage logging in NannyNL
    • Versions
      • Version 0.22.0
      • Version 0.21.0
Powered by GitBook
On this page
  • The model and dataset
  • Reference and analysis sets
  • Monitoring with nannyML Cloud
  • Step 1: Add a new model
  • Step 2: Define the problem type and main metric
  • Step 3: Configure the Reference set
  • Step 4: Define the reference dataset schema
  • Step 5: Configure the Analysis set
  • Step 6: Start monitoring
  1. Model Monitoring
  2. Tutorials

Monitoring a text classification model

Tutorial explaining how to monitor text classification models with NannyML

PreviousMonitoring with segmentationNextMonitoring a computer vision model

In this tutorial, we will use nannyML cloud to monitor a sentiment analysis text classification model where the model's goal is to predict the sentiment (Negative, Neutral, Positive) of a review left on Amazon.

The model and dataset

We will use a model trained on a subset of the . The trained model can be found in the .

For details of how this model was produced, check out the blog post: .

Reference and analysis sets

To evaluate the model in production, we have two sets:

  • Reference set - which contains all model inputs along with the model’s predictions and labels. This set establishes a baseline for every metric we want to monitor. Find the reference set:

  • Analysis set - which contains all model inputs extracted from a production set with the model’s prediction, and in this case, labels. The analysis set is where NannyML analyzes/monitors the model’s performance and data drift of the model using the knowledge gained from the reference set. Find the analysis set:

Monitoring with nannyML Cloud

Step 1: Add a new model

Click the Add model button to create a new model on your nannyML cloud dashboard.

Step 2: Define the problem type and main metric

Each review that we are classifying can be Negative, Positive, or Neutral. For this reason, we will set the problem type as Multiclass classification.

We will be monitoring the model's F1-score on a weekly basis.

Step 3: Configure the Reference set

Select "Upload via public link".

Step 4: Define the reference dataset schema

  1. Select the column timestamp as the Timestamp column

  2. Select the column predicted_sentiment as the Prediction column

  3. Select the real_sentiment as the Target column

  4. Flag the columns negative_sentiment_pred_proba, neutral_sentiment_pred_proba, positive_sentiment_pred_proba as Prediction Scores.

Step 5: Configure the Analysis set

Select "Upload via public link".

Step 6: Start monitoring

Use the following public URL to link the Reference dataset:

Use the following public URL to link the Analysis dataset:

https://raw.githubusercontent.com/NannyML/sample_datasets/main/amazon_reviews/amazon_reviews_reference.csv
https://raw.githubusercontent.com/NannyML/sample_datasets/main/amazon_reviews/amazon_reviews_analysis_targets.csv
Multilingual Amazon Reviews Dataset
nannyML's hugging face hub
Are your NLP models deteriorating post-deployment? Let’s use unlabelled data to find out
https://raw.githubusercontent.com/NannyML/sample_datasets/main/amazon_reviews/amazon_reviews_reference.csv
https://raw.githubusercontent.com/NannyML/sample_datasets/main/amazon_reviews/amazon_reviews_analysis_targets.csv