Rigorous and Practical Server Design Evaluation – tutorial at ASPLOS 2014
Submitted by Djordje Jevdjic
The emergence of cloud computing as a dominant computing platform highlights
the need for practical and rigorous architectural evaluation of server systems.
Such evaluation mandates the use of a variety of real-world server workloads,
all of which are radically different from traditional desktop and scientific
benchmarks. Unfortunately, deep and complex software stacks of both conventional
(e.g., OLTP, DSS) and emerging scale-out (e.g., Media Streaming, Web Search)
server workloads make the evaluation process even harder and slower, postponing
the adoption of realistic server benchmarks within the architectural community.
CloudSuite is an on-going effort towards a common architectural evaluation
basis, aimed at providing an up-to-date suite of benchmarks that represent
popular scale-out cloud applications commonly found in today’s datacenters.
The latest version, CloudSuite 2.0, comprises of eight scale-out applications
that feature real-world server software stacks and datasets that can be used as
realistic benchmarks for the scale-out applications class. These applications
are data analytics, data caching, data serving, graph processing, media
streaming, software testing, web search, and web serving. CloudSuite provides
the means for throughput and quality of service (QoS) measurement for each
In this tutorial, we first introduce CloudSuite 2.0 and demonstrate how to run
the benchmarks on real hardware. As evaluating server software stacks via
full-system simulation is prohibitively slow, a need for statistical sampling
emerges. A rigorous sampling methodology is of utmost importance, because
improper sampling can lead to highly misleading results. Devising the right
sampling methodology for server workloads is not trivial. For that reason,
our tutorial covers the SMARTS sampling methodology, a widely applicable
framework that enables fast cycle-accurate simulation of both conventional and
emerging server workloads on various full-system architectural simulators. We
demonstrate how the SMARTS methodology can be applied in today’s simulators
through a case study; we describe the complete simulation process using Flexus,
a full-system multiprocessor simulator that leverages the statistical sampling
methodology for fast, yet rigorous evaluation of CloudSuite 2.0 benchmarks.
Our tutorial will take place on Sunday, March 2nd, in conjunction with ASPLOS
2014 in Salt Lake City. For further information on the tutorial, please visit
our website at http://parsa.epfl.ch/cloudsuite/asplos14-tutorial or send us
an e-mail to firstname.lastname@example.org.