Operational experience running Hadoop XRootD Fallback

[摘要] In April of 2014, the UCSD T2 Center deployed hdfs-xrootd-fallback, a UCSD- developed software system that interfaces Hadoop with XRootD to increase reliability of the Hadoop file system. The hdfs-xrootd-fallback system allows a site to depend less on local file replication and more on global replication provided by the XRootD federation to ensure data redundancy. Deploying the software has allowed us to reduce Hadoop replication on a significant subset of files in our cluster, freeing hundreds of terabytes in our local storage, and to recover HDFS blocks lost due to storage degradation. An overview of the architecture of the hdfs-xrootd-fallback system will be presented, as well as details of our experience operating the service over the past year.

[发布机构] University of California, 9500 Gilman Dr, San Diego; CA; 92093, United States^1

[效力级别] 计算机科学 [学科分类] 计算机科学（综合）

[关键词] Data redundancy;File replication;File systems;Operational experience;Software systems