Tech Showcase: Prometheus: Co-Designing Distributed Systems and Datacenter Hardware

The modern datacenter fabric increasingly includes programmable devices capable of increasingly sophisticated processing – ranging from programmable switches that support in-network computation to network-attached FPGAs and smart storage devices. This project asks how we can leverage this new capability to build a new generation of storage and data processing systems that achieve dramatically better performance, reliability, and efficiency.

The key idea is to co-design distributed systems with the data center hardware: introducing new low-level primitives and building systems around them. We have already used this approach to achieve order-of-magnitude speedups for replicated services and transaction coordination. Future directions include: a scalable analytics system that handles massive volumes of network telemetry data by moving the first-level filtering and aggregation to programmable network devices; in-network aggregation for faster training of deep neural networks; and replicated remote storage with minimal CPU overhead by using network-attached programmable solid-state drives.

Series: Microsoft Research Faculty Summit