A Design Pattern for Deploying Machine Learning Models to Production

Machine Learning (ML) becomes increasingly popular; industry spends billions of dollars building ML systems. Data scientists have come up with many good algorithms and trained models. However, putting those ML models into production is still in the early stage. The deployment process is distinct from that for traditional software applications; it is not yet well understood among data scientists and IT engineers in their roles and responsibilities, resulting in many anti-pattern practices [21]. The key issues identified by researchers at Google[40] include lack of production-like prototyping stack for data scientists, monolithic programs not fitted for component based ML system orchestration, and lack of best practices in system design. To find solutions, teams need to understand the inherent structure of ML systems and to find ML engineering best practices. This paper presents an abstraction of ML system design process, a design pattern named Model-Service-Client + Retraining (MSC/R) consisting of four main components: Model (data and trained model), Service (model serving infrastructure), Client (user interface), and Retraining (model monitoring and retraining). Data scientists and engineers can use this pattern as a discipline in designing and deploying ML pipelines methodically. They can separate concerns, modularize ML systems, and work in parallel. This paper also gives case studies on how to use MSC/R to quickly and reliably deploy two ML models -- YOLOv3, an object detection model, and Stock Prediction using Long Short-Term Memory (LSTM) algorithm onto AWS and GCP clouds. Two different implementation approaches are used: serving the model as a microservice RESTful API on AWS managed container platform ECS, and on GCP serverless platform Cloud Run. In the end, this paper gives analysis and discussion on how using the MSC/R design pattern helps to meet the objectives of implementing ML production systems and solve the common problems. It also provides insights and recommendations.