Scale-up vs Scale-out for Hadoop: Time to rethink? - Appuswam, Raja; Gkantsidis, Christos; Narayanan, Dushyanth; Hodson, Orion; Rowstron, Ant (ACM Symposium on Cloud Computing, 2013)
n the last decade we have seen a huge deployment of cheap clusters to run data analytics workloads. The conventional wisdom in industry and academia is that scaling out using a cluster of commodity machines is better for these workloads than scaling up by adding more resources to a single server. Popular analytics infrastructures such as Hadoop are aimed at such a cluster scale-out environment. Is this the right approach? Our measurements as well as other recent work shows that the majority of real-world analytic jobs process less than 100 GB of input, but popular infrastructures such as Hadoop/MapReduce were originally designed for petascale processing. We claim that a single “scale-up” server can process each of these jobs and do as well or better than a cluster in terms of performance, cost, power, and server density. We present an evaluation across 11 representative Hadoop jobs that shows scale-up to be competitive in all cases and signiﬁcantly better in some cases, than scale-out. To achieve that performance, we describe several modiﬁcations to the Hadoop runtime that target scale-up conﬁguration. These changes are transparent, do not require any changes to application code, and do not compromise scale-out performance; at the same time our evaluation shows that they do signiﬁcantly improve Hadoop’s scale-up performance.