A scalable architecture for online anomaly detection of WLCG batch jobs

[摘要] For data centres it is increasingly important to monitor the network usage, and learn from network usage patterns. Especially configuration issues or misbehaving batch jobs preventing a smooth operation need to be detected as early as possible. At the GridKa data and computing centre we therefore operate a tool BPNetMon for monitoring traffic data and characteristics of WLCG batch jobs and pilots locally on different worker nodes. On the one hand local information itself are not sufficient to detect anomalies for several reasons, e.g. the underlying job distribution on a single worker node might change or there might be a local misconfiguration. On the other hand a centralised anomaly detection approach does not scale regarding network communication as well as computational costs. We therefore propose a scalable architecture based on concepts of a super-peer network.

[发布机构] Karlsruhe Institute of Technology, Steinbuch Centre for Computing, Hermann-von-Helmholtz-Platz 1, Eggenstein-Leopoldshafen; 76344, Germany^1

[效力级别] 计算机科学 [学科分类] 计算机科学（综合）

[关键词] Anomaly detection;Computational costs;Local information;Misconfigurations;Network communications;Online anomaly detection;Scalable architectures;Superpeer networks