Developing ML solutions
Fundamental Concepts of MLOps
MLOps applies DevOps practices to machine learning. It operationalizes the whole lifecycle so models are developed, deployed, monitored, and retrained systematically and repeatedly. It combines people, process, and technology.
ML needs its own operations because models degrade as data changes. A model that works at launch can get worse over weeks or months.
Goals
- Speed up the lifecycle through automation
- Improve quality through testing and monitoring
- Encourage collaboration between data scientists, data engineers, software engineers, and IT operations
- Provide transparency, explainability, auditability, and security through model governance
Key principles
- Version control of data, code, and models, for reproducibility and rollback
- Automation of ingestion, preprocessing, training, validation, and deployment
- CI/CD for ML:
- Continuous integration: tests code, data, and models
- Continuous delivery: deploys new models automatically
- Continuous training: retrains models automatically
- Continuous monitoring: tracks data, models, and business metrics
- Model governance: documentation, review and approval before deployment (checking fairness and bias), data protection, and compliance
A production ML setup usually has separate model build (training) and deployment pipelines. The build pipeline runs, for example, when new data arrives.
AWS services for MLOps
- Prepare data: SageMaker Data Wrangler (low-code) or the SageMaker Processing API
- Store features: SageMaker Feature Store
- Train and tune: SageMaker training and automatic model tuning
- Track experiments: SageMaker Experiments
- Register models: SageMaker Model Registry
- Orchestrate: SageMaker Pipelines
- Monitor: SageMaker Model Monitor