Deep Learning Inference Service at Microsoft - Soifer, Jonathan; Li, Jason; Li, Mingqin; Zhu, Jeffrey; Li, Yingnan; He, Yuxiong; Zheng, Elton; Oltean, Adi; Mosyak, Maya; Barnes, Chris; Liu, Thomas; Wang, Junhua (2019 USENIX Conference on Operational Machine Learning (OpML ’19), 2019)
This paper introduces the Deep Learning Inference Service, an online production service at Microsoft for ultra-low-latency deep neural network model inference. We present the system architecture and deep dive into core concepts such as intelligent model placement, heterogeneous resource management, resource isolation, and efficient routing. We also present production scale and performance numbers.