Secure, Efficient Data Transport and Replica Management for High-Performance Data-Intensive Computing

By William E. Allcock, J. Bester, John Bresnahan, Ann L. Chervenak, Carl Kesselman, Sam Meder, Veronika Nefedova, D. Quesnel, Steven Tuecke, Ian T Foster. 2001 Eighteenth IEEE Symposium on Mass Storage Systems and Technologies, 2001

326 citations recorded by the source corpus.

An emerging class of data-intensive applications involve the geographically dispersed extraction of complex scientific information from very large collections of measured or computed data. Such applications arise, for example, in experimental physics, where the data in question is generated by accelerators, and in simulation science, where the data is generated by supercomputers. So-called Data Grids provide essential infrastructure for such applications, much as the Internet provides essential services for applications such as e-mail and the Web. We describe here two services that we believe are fundamental to any Data Grid: reliable, high-speed transport and replica management. Our high-speed transport service, GridFTP, extends the popular FTP protocol with new features required for Data Grid applications, such as striping and partial file access. Our replica management service integrates a replica catalog with GridFTP transfers to provide for the creation, registration, location, and management of dataset replicas. We present the design of both services and also preliminary performance results. Our implementations exploit security and other services provided by the Globus Toolkit.

Subfields: Computer Networks and Communications

View Paper: https://www.semanticscholar.org/paper/b942251fc76dc1b29b62fcb29b68054cd48867f5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

Citations in the corpus

Mapping Abstract Complex Workflows onto Grid Environments (2003, background): “In terms of data management, Globus provides information about locating replicas of files (via RLS [13]) and the means of highperformance data transfer (via GridFTP [2]).”
Chimera: a virtual data system for representing, querying, and automating data derivation (2002, methodology): “All data transfers used the Grid-enabled data transfer tool GridFTP [2].”
Pegasus: A framework for mapping complex scientific workflows onto distributed systems (2005, background): “GridFTP allows for high-performance , secure data transfer in the wide area net- works.”
A case for grid computing on virtual machines (2003, methodology): “Virtual machine sessions are instantiated using globusrun (Globus 2 .0 toolkit) within a LAN.”
A taxonomy of Data Grids for distributed data sharing, management, and processing (2005, methodology): “The catalog can be organized as a tree as in the case of LDAP-based (Lightweight Directory Access Protocol) catalogs such as the Globus Replica Catalog [Allcock et al. 2001].”
The Anatomy of the Grid: Enabling Scalable Virtual Organizations (2001, methodology): “• Data replication services support the management of VO storage (and perhaps also network and computing) resources to maximize data access performance with respect to metrics such as response time, reliability, and cost (Allcock et al., 2001; Hoschek et al., 2000).”
Stork: making data placement a first class citizen in the grid (2004, background): “After letting Stork use the alternative protocol (in this case GridFTP) to perform the transfers for a while, we restarted the DiskRouter server at the SDSC site.”
A community authorization service for group collaboration (2002, background): “The replica manager [ 30 ] maps from logical filenames to physical locations (i.e., hostnames and paths).”