Reliable maintenance of machine learning models

Hello!

@juliasilge

@juliasilge@fosstodon.org

youtube.com/juliasilge

juliasilge.com

Maintaining ML models is NOTHING LIKE THIS

Never-ending laundry process

Maintaining ML models is never done

New load of laundry – when and how to retrain model

All software products need to be maintained, but models are both a software artifact and a statistical artifact

Both software and statistical products

Unlike something that is “only” a software product, you could, say, retrain your model with new data and change its statistical properties without changing its software properties

What does performance mean?

When we talk about software or statistical properties, we’re really talking about performance

One of the most basic prerequisites for model maintenance is monitoring the model’s performance

Measuring model performance allows us to say:

My model is performing well!

👩🏼‍🔧 My model returns predictions quickly, doesn’t use too much memory or processing power, and doesn’t have outages.

Metrics

• latency
• memory and CPU usage
• uptime

My model is performing well!

👩🏽‍🔬 My model returns predictions that are close to the true values for the predicted quantity.

Metrics

• accuracy
• ROC AUC
• F1 score
• RMSE
• log loss

As data practitioners, we focus on monitoring statistical performance

Failures in statistical performance can be silent

Statistical properties of a model have unique tendencies toward silence

MODEL DRIFT

data drift + concept drift

DATA DRIFT

Let’s say I run a laundry service, and I have built a model to predict someone’s likelihood to become my customer

I use a bunch of predictors, like how many loads of laundry they need to do each month, their income, whether they live in a house or apartment, and a bunch of other things

Data drift is about drift or change in these inputs, compared to what we had when we trained the model.

Very concretely, we measure this through summary statistics, through visualization

Monitor your inputs

Monitor your inputs

We can make this kind of comparison visually, or we can use a statistical test

Monitor your inputs

True/real changes in distribution vs. changes in data collection or similar

DATA DRIFT

Monitor your inputs

CONCEPT DRIFT

Monitor your outputs

Relationship between the input and output is changing over time

In the city where I run a laundry service, the new apartments that are being built all have laundry in people’s units

Monitor your outputs

library(vetiver)

laundry_service_monitoring |> 
  vetiver_compute_metrics(date, "week", customer, .pred)
#> # A tibble: 30 × 5
#>    .index        .n .metric  .estimator .estimate
#>    <date>     <int> <chr>    <chr>          <dbl>
#>  1 2023-03-05    14 accuracy binary         0.857
#>  2 2023-03-05    14 kap      binary         0.708
#>  3 2023-03-09    34 accuracy binary         0.882
#>  4 2023-03-09    34 kap      binary         0.767
#>  5 2023-03-16    25 accuracy binary         0.8  
#>  6 2023-03-16    25 kap      binary         0.525
#>  7 2023-03-23    32 accuracy binary         0.844
#>  8 2023-03-23    32 kap      binary         0.685
#>  9 2023-03-30    36 accuracy binary         0.806
#> 10 2023-03-30    36 kap      binary         0.611
#> # ℹ 20 more rows

Monitor your outputs

library(vetiver)

laundry_service_monitoring |> 
  vetiver_compute_metrics(date, "week", customer, .pred) |> 
  vetiver_plot_metrics()

Feedback loops 🔁

Deployment of an ML model may cause data and/or concept drift

Examples

• Movie recommendation systems on Netflix, Disney+, Hulu
• Identifying fraudulent credit card transactions at Stripe
• Predictive policing models

Users take some action as a result of a prediction, users rate or correct the quality of a prediction, produce feedback automatically

Stages of model monitoring maturity

1. Manual 🙂

2. Reproducible 🤓

3. Automated 🤩

Resilient models that are successful in the long term

We’ve talked about

• understanding what performance means for models
• how to consider both the software and statistical characteristics of a model
• measuring inputs and outputs to get a handle on drift

These are what you need to have a model that is easier to maintain

Learn more

• Documentation at https://vetiver.rstudio.com/

• Webinar by Isabel Zimmerman and me for Posit Enterprise Meetup

• End-to-end demos from Posit Solution Engineering in R and Python

Post questions at pos.it/slido-CD 🎯

Thank you!

@juliasilge

@juliasilge@fosstodon.org

youtube.com/juliasilge

juliasilge.com