Adding a model
Last updated
Last updated
Two methods are available to add a model to NannyML Cloud. The optimal way is to do it programmatically, which can be found on the page. Alternatively, you can manually add the model using the NannyML Cloud UI, which we'll explain here.
If you prefer a video walkthrough and you upload your data from Azure, here's our YouTube guide:
You need to provide four pieces of information about your model:
The machine learning problem type The type of problem the machine learning model is dealing with. The different options are binary classification, multiclass classification, and regression. This has lots of implications. It impacts what type of model output and target data NannyML is expecting and which metrics NannyML can calculate. This cannot be changed later.
The main performance metric Depending on which problem type you select, the available metrics will change. These metrics can always be changed later, and you can monitor multiple metrics simultaneously. Currently, we support the following metrics:
ROC-AUC
F1
Precision
Recall
Specificity
Accuracy
Business value
Confusion matrix elements
The reference dataset is the dataset NannyML will use as a baseline for monitoring your model. This dataset ideally represents a time when the model worked as expected. The ideal candidate for this is the test set. You need to point NannyML to where this dataset is located and provide some basic information about the dataset schema.
Point nannyML to the reference dataset location
Pick one of the following upload options:
Provide a public URL
If the dataset is accessible via a public URL, you can provide that link here:
To try out NannyML, use one of our public datasets on GitHub. Here is a link to the synthetic car price prediction - reference dataset:
Provide reference dataset information
NannyML requires schema information about the reference dataset. While it automatically gets column details, it's always wise to double-check. The most critical columns to define are listed on the left. The columns you need to specify depend on the type of machine-learning problem you chose at the beginning of this workflow. All other columns are automatically treated as features. Additionally, NannyML automatically detects the data types of these feature columns.
The following columns have to be specified:
Timestamp This provides NannyML with the date and time that the prediction was made.
Prediction The model output that the model predicted for its target outcome.
Target The ground truth or actual outcome of what the model is predicting.
Identifier A unique identifier for each row in the dataset. NannyML will use this column to join analysis and target data sources.
The mapping of the columns can be changed when scrolling horizontally. It is possible to ignore specific columns or flag columns that should be used for joining predictions and targets later.
The analysis dataset is what NannyML uses to analyze the performance of the monitored model. Typically, it will consist of the latest production data up to a desired point in the past, which should be after the reference dataset ends.
Point nannyML to the analysis dataset location
Pick one of the following options:
Provide a public URL
If the dataset is accessible via a public URL, you can provide that link here:
To try out NannyML, you can use one of our public datasets on GitHub. Here is a link to the synthetic car price prediction-analysis dataset:
This step is only necessary when targets are not part of the analysis dataset and when they are available.
Point nannyML to the target dataset location
Pick one of the following options:
Provide a public URL
If the dataset is accessible via a public URL, you can provide that link here:
To try out NannyML, you can use one of our public datasets on Git Hub. Here is a link to the synthetic car price prediction-analysis target dataset:
Review your model settings and start monitoring! 🚀
How the data has to be chunked Chunking determines how metrics will be aggregated, i.e. the granularity of the monitoring analysis. The options are either time-based or size-based. For time-based chunking the options are daily, weekly, monthly, quarterly, and yearly chunking. For size-based chunking you can select a chunk size; i.e. a number of records to have in a single chunk. The last chunk may not be completely "filled" if there are not enough records. It will be recomputed automatically as more records are added and the chunk "fills up". The chunking unit can always be changed later in the .
We currently only support time-based and size-based chunking; if you need support for number-based chunking, .
The Account key is a secret key that gives access to all the files in the storage account. It can be found through the Azure portal. .
The Sas Token is a temporary token that allows NannyML to impersonate the user. It has to be specifically created when doing the onboarding. .
Not currently available in the product. For custom support, please .
The Account key is a secret key that gives access to all the files in the storage account. It can be found through the Azure portal. .
The Sas Token is a temporary token that allows NannyML to impersonate the user. It has to be specifically created when doing the onboarding. .
Not currently available in the product. For custom support, please .
The Account key is a secret key that gives access to all the files in the storage account. It can be found through the Azure portal. .
The Sas Token is a temporary token that allows NannyML to impersonate the user. It has to specifically be created when doing the onboarding. .
Not currently available in the product. For custom support, please .
