The ATLAS Event Service: A new approach to event processing

[摘要] The ATLAS Event Service (ES) implements a new fine grained approach to HEP event processing, designed to be agile and efficient in exploiting transient, short-lived resources such as HPC hole-filling, spot market commercial clouds, and volunteer computing. Input and output control and data flows, bookkeeping, monitoring, and data storage are all managed at the event level in an implementation capable of supporting ATLAS-scale distributed processing throughputs (about 4M CPU-hours/day). Input data flows utilize remote data repositories with no data locality or pre-staging requirements, minimizing the use of costly storage in favor of strongly leveraging powerful networks. Object stores provide a highly scalable means of remotely storing the quasi-continuous, fine grained outputs that give ES based applications a very light data footprint on a processing resource, and ensure negligible losses should the resource suddenly vanish. We will describe the motivations for the ES system, its unique features and capabilities, its architecture and the highly scalable tools and technologies employed in its implementation, and its applications in ATLAS processing on HPCs, commercial cloud resources, volunteer computing, and grid resources. Notice: This manuscript has been authored by employees of Brookhaven Science Associates, LLC under Contract No. DE-AC02-98CH10886 with the U.S. Department of Energy. The publisher by accepting the manuscript for publication acknowledges that the United States Government retains a non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes.

[发布机构] Lawrence Berkeley National Laboratory, 1 Cyclotron Rd, Berkeley; CA; 94720, United States^1;University of Texas at Arlington, 701 S Nedderman Dr, Arlington; TX; 76019, United States^2;University of Wisconsin - Madison, Madison; WI; 53706, United States^3;Brookhaven National Laboratory, Upton; NY; 11973, United States^4;Argonne National Laboratory, 9700 South Cass Avenue, Lemont; IL; 60439, United States^5

[效力级别] 计算机科学 [学科分类] 计算机科学（综合）

[关键词] Distributed processing;Event Processing;Input and outputs;ITS applications;Processing resources;Tools and technologies;U.S. Department of Energy;Volunteer computing