Introduction Notes

Page 1

Introduction

Intro

ML code is often the smallest part of an ML solution

Most revolves around data (pipelines, validation, feature extraction, etc)

MLOps = a set of processes and automation for managing code, data, and models to improve performance, stability and long-term efficiency of ML systems

MLOps = DevOps + DataOps + ModelOps

Code, data, and model all change overtime and the ML pipeline needs to be adjusted accordingly

Why we should care about MLOps

MLOps helps reduce risk

Technical risk (bad models, poor design)

Compliance risk

Improved efficiency via automation

Less error prone

Databricks manages MLOps systems using various tools

DevOps via Github type softwares

DataOps via Delta Lake and Feature Store

ModelOps via mlFlow

4 guiding principles

Always keep your business goals in mind

Take a data-centric approach to machine-learning

Implement MLOps in a modular fashion

Process should guide automation

For this class we will focus on mlflow, feature store, and delta lake

Managing machine learning lifecycle means handling the following

Reproducibility of data

Reproducibility of code

Reproducibility of models

Automated integration with prod systems

Mlflow has 4 major components

Mlflow tracking

Mlflow projects

Mlflow models

Mlflow model registry

Full ML Lifecycle

Data scientists build features and data engineers provide infra for automating that

Data scientists build models and log them to mlflow, recording environment info

Data scientists move models to staging

Deployment engineers manage CI/CD tools which promote models to production

Page 2