DataLight: data transfer and logging of large output applications in Grid environments

Abstract

In this paper we present a powerful and lightweight library called DataLight that handles the transfer of large data file sets from the local storage of the Worker Node to an gLite Storage Element. The library is very easy to integrate into non-Grid/legacy code and it is flexible enough to allow for overlapping of several data transfer and computation. It has the ability to send messages to a Web Portal while the application is running, so that a user can login to the Portal and receive immediate updates. In particular the user can start post-processing the produced data while the application is still running. We present the application details both of the library and Portal and discuss some relevant security issues.