Inter-datacenter bulk transfers with netstitcher

By Nikolaos Laoutaris, Michael Sirivianos, Xiaoyuan Yang, Pablo Rodríguez Rodríguez. Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication, 2011

315 citations recorded by the source corpus.

Large datacenter operators with sites at multiple locations dimension their key resources according to the peak demand of the geographic area that each site covers. The demand of specific areas follows strong diurnal patterns with high peak to valley ratios that result in poor average utilization across a day. In this paper, we show how to rescue unutilized bandwidth across multiple datacenters and backbone networks and use it for non-real-time applications, such as backups, propagation of bulky updates, and migration of data. Achieving the above is non-trivial since leftover bandwidth appears at different times, for different durations, and at different places in the world. For this purpose, we have designed, implemented, and validated NetStitcher, a system that employs a network of storage nodes to stitch together unutilized bandwidth, whenever and wherever it exists. It gathers information about leftover resources, uses a store-and-forward algorithm to schedule data transfers, and adapts to resource fluctuations. We have compared NetStitcher with other bulk transfer mechanisms using both a testbed and a live deployment on a real CDN. Our testbed evaluation shows that NetStitcher outperforms all other mechanisms and can rescue up to five times additional datacenter bandwidth thus making it a valuable tool for datacenter providers. Our live CDN deployment demonstrates that our solution can perform large data transfers at a much lower cost than naive end-to-end or store-and-forward schemes.

Subfields: Computer Networks and Communications, Information Systems

View Paper: https://www.semanticscholar.org/paper/81547b4b26e14a9c1ff8a71198252542ee459377

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

Citations in the corpus

Achieving high utilization with software-driven WAN (2013, background): “Better knowledge of such needs can further boost eﬃciency, for instance, by enabling store-and-forward transfers through intermediate DCs [21].”
Gaia: Geo-Distributed Machine Learning Approaching LAN Speeds (2017, background): “While ML algorithms have different types across different domains, almost all have the same goal—searching for the best model (usually a set of parameters) to describe or explain the input data [77].”
Low Latency Geo-distributed Data Analytics (2015, background): “Other works optimize data placement to improve WAN latencies and utilization [41, 50].”