# Getting Probability Distribution of Performance Metric when some observations have labels

In some cases experiment data can contain both: observations with labels and observations without labels. This happens for example when the predictions of the model affect whether we get to observe the label or not. For example, in credit scoring, we will never see the label for credit applicants who were rejected. This is an example of censored confusion matrix - specific elements of the confusion matrix are not available (in the credit scoring case we never observe true and false negatives). 

For such cases we would still want to get single performance metric probability distribution that accounts for both: observations with and without labels. NannyML does that by separately calculating posteriors for these two types of observations and combining them into single posterior using Bayes theorem. Denoting $$p(m|labeled, unlabeled)$$ a posterior of performance metric given labelled and unlabeled observations we write:

$$
p(m|labeled, unlabeled)  \propto p(labeled|m)\times p(m|unlabeled)
$$

The $$p(m|unlabeled)$$ term is the posterior [we get from observations without labels](/cloud/v0.24.0/probabilistic-model-evaluation/how-it-works/getting-probability-distribution-of-performance-metric-without-targets.md). The $$p(labeled|m)$$is the likelihood of data with labels given the metric value of interest. We can plug it in directly (for metrics like *accuracy* this is the binomial function) or use the [posterior for that we get for the data with labels](/cloud/v0.24.0/probabilistic-model-evaluation/how-it-works/getting-probability-distribution-of-a-performance-metric-with-targets.md). This is possible, because for uniform priors $$p(m|labeled)  \propto p(labeled|m)$$.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.nannyml.com/cloud/v0.24.0/probabilistic-model-evaluation/how-it-works/getting-probability-distribution-of-performance-metric-when-some-observations-have-labels.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.