Analyzing Request Volatility in Cloud-Based Machine Learning: A Review of Alibaba's Machine Learning as a Service Platform

Abstract

Cloud ML (MLaaS) services need to schedule mixed training, pre-processing and inference requests arriving by different processes 
for different users, models, tasks and time scales. This paper reviews the literature on Alibaba's Platform for AI (PAI), large GPU clusters, traces of cloud storage, serverless workloads, traffic self-similarity and elastic deep-learning serving, in order to clarify how request volatility should be quantified and translated into operation. The literature finds that simple means and memoryless inter-arrival distributions are inadequate: cloud ML traffic can combine abrupt bursts, long-range dependence, cross-layer amplification, resource-demand skewness and colocated job interference. The paper therefore develops a reproducible analytical framework based on multi-resolution aggregation, coefficient of variation, Fano factor, burstiness index, autocorrelation, variance-time analysis, Hurst calculation, peak-episode detection and workload-resource correlation.These metrics are related to a practical web-based system based on trace ingestion, pre-processing, MySQL store, a volatility analytics engine, dashboards, report generation and inference-based resource recommendations, with no performance gains claimed in code, data or procedures. The key conclusion is that volatility must be jointly assessed across hierarchy, time scale and resource type prior to choosing autoscalers or schedulers.