A data placement strategy in scientific cloud workflows

By Dong Yuan, Yun Yang, Xiao Liu, Jinjun Chen. Future generations computer systems, 2010

351 citations recorded by the source corpus.

In scientific cloud workflows, large amounts of application data need to be stored in distributed data centres. To effectively store these data, a data manager must intelligently select data centres in which these data will reside. This is, however, not the case for data which must have a fixed location. When one task needs several datasets located in different data centres, the movement of large volumes of data becomes a challenge. In this paper, we propose a matrix based k-means clustering strategy for data placement in scientific cloud workflows. The strategy contains two algorithms that group the existing datasets in k data centres during the workflow build-time stage, and dynamically clusters newly generated datasets to the most appropriate data centres-based on dependencies-during the runtime stage. Simulations show that our algorithm can effectively reduce data movement during the workflow's execution.

Subfields: Computer Networks and Communications, Information Systems, Information Systems and Management

View Paper: https://www.semanticscholar.org/paper/b2823aada0d6c25e13bca5254df5b5ad49883429