Karteikarten
Data Engineering Lecture 9: MLOps
Karten in diesem Stapel (33)
- 1MLOps is right at the center of- Data Engineering
- Machine Learning
- Software Engineering - 2Data engineering- To provide a consistent data flow to enable data-driven work
- The aspect of data science that focuses on the practical application of data collection and analysis
- Data engineering can be thought of as a superset of business intelligence and data warehousing i.e., setting up and organizing the organization's’ data infrastructure - 3Machine Learning- The branch of AI and CS which focuses on the use of data and algorithms to imitate the way that humans learn
- An application of AI that provides systems the ability to automatically learn and improve from experience without being explicitly programmed
- A type of AI that allows software applications to become more accurate at predicting outcomes without being explicitly programmed to do so - 4Software Engineering- The process of analyzing user requirements and then designing, building, and testing software applications which will satisfy those requirements
- Programming integrated over time
- A detailed study of engineering to the design, development, and maintenance of software - 5DevOpsProcess whose aim is to shorten the system development lifecycle and provide continuous delivery with software of high quality.
- 6Can we do the same for ML applications?MLOps was born!
- 7MLOpsWhen the DevOps process is explicitly applied to ML
- 8However, directly applying DevOps to ML doesn’t work. 4 pillars where the two differ:- Complexity
- Continuous Integration
- Continuous Deployment
- Continuous Training - 9Complexity - ML models cannot be treated like traditional software artifacts, because ML for production is more complex, we expect changes in:1. The code &
2. The model &
3. The (input) data.
▪ This makes it (1) harder to test, (2) harder to explain (3) harder to improve.
▪ MLOps should be more complex than DevOps. - 10CIThe practice of regularly merging all working copies into a shared system.
- 11CI - Work1. write tests
2. execute tests (automatically)
3. merge - 12CI - BenefitTests can be used indefinitely
- 13CI - GoalWe write tests to prevent unexpected behavior and failures after merging the working copies, such that the software becomes more robust.
- 14CI - ProblemTraditional testing doesn’t work for ML.
▪ reusing the same test cases repeatedly can introduce risk of overfitting
▪ every pass/fail signal leaks information about the test set itself - 15Example▪ Test assertion: n - o > 0.01
▪ Test set: finite set of i.i.d. samples from the underlying data distribution.
▪ What happens if we test with test set size 1? Size 400?
▪ Problem: Testing in ML is not deterministic, it’s an estimate, it is inherently uncertain
▪ Solution: Interpret the evaluation result from a probabilistic view
▪ Implementation: Test with evaluative metrics, confidence intervals and tolerance levels - 16Continuous deployment (CD)An extension of CI, further testing and deploying to staging and/or production
- 17CD - Work1. Write tests and deployment pipeline(s)
2. Execute tests, then
3. Deploy and run more tests - 18CD - benefitApplication can be deployed in minutes
- 19CD - goalWe write tests to prevent unwanted behavior and failures in the staging and production environments, write pipelines to deliver the software quickly.
- 20CD - ProblemTypes of tests used in DevOps are inadequate and provide too little information:
▪ Where did we fail? E.g., we input a giraffe in a model for classifying cats / dogs
▪ Data and concept drift - 21CD - SolutionExtend CD principles from DevOps, by allowing a MLOps system to automatically retire models and gather data throughout the system
- 22CD - ImplementationContinuously monitor a model’s performance and input data, and store this information for analysis
- 23Continuous training▪ Automated (re-)training of models when a specific event happens
▪ Example events:
▪ New training data (i.e., statistical properties, more data)
▪ Model performance deterioration (i.e., monitoring the trend)
▪ Implementation: Introduce (reusable) ML pipelines for training and experimentation
▪ Automatically trigger when:
▪ New training data
▪ Model performance deterioration - 24Continous training solution
- 25Feature StoreContains cleaned data in the right shape, usable for training. The ML pipeline is executed with the new data. All cleaning is done already, minor changes possible
- 26We want- Automated (re-)training of models when a specific event occurs.
- To store information about the data, training, experimentation, the model, our code, etc.
▪ In other words, we need to learn from our mistakes by extracting metadata from training, evaluating, and validating steps of the pipeline (e.g., hyperparameters).
▪ Which model configurations succeed? which failed? what needs tweaking? - 27Store and version control source code▪ Reusable pipeline & reproducible experiments
▪ All code, environment info & dependencies need to be version controlled. - 28Store immutable feature set from the feature storeReproducible experiments, the feature set should be immutable and uniquely identifiable
- 29All in all, the components areAutomated preparation (load & store transformed data) execution of ML pipelines (extraction, preparation, training, evaluation, validation) and managing artefacts (metadata, source code & feature set)
- 30In summary, DevOps for ML1. ML applications are complex.
2. Test with evaluative metrics, confidence intervals and tolerance levels.
3. Give high quality data a central role in the system.
4. Monitor models and input data.
5. Store, manage and version-control artifacts. - 31More additions for applying DevOps to ML- Storage
▪ Metadata store
▪ Raw data store
▪ Feature store
- Additional storage
▪ Model registry
▪ Source code repository
- Services
▪ Data ingestion service – user can upload new data
- Additional service
▪ Inference service – to provide the users the model in production - 32The MLOps landscape- Has not matured
- Is prone to change
▪ Projects have different needs, it’s hard to generalize.
▪ The maturity of a MLOps system has many levels. To what extent do we want to automate things?
- Building an MLOps system
▪ End-to-end solutions
▪ Amazon Sagemaker
▪ Microsoft Azure MLOps suite
▪ Google Cloud MLOps suite
▪ Custom-built (i.e., using existing tools or create your own) - 33The MLOps landscape example
Informationen
- Beschreibung
- Engineering Karteikarten am Data Engineering Lecture 9: MLOps, erstellt von Tom van Eijk am 21/12/2021.
- Erstellt von
- Tom van Eijk
- Aufrufe
- 1
- Bewertung