Analysis and improvements of behaviour-based malware detection mechanisms

[摘要] The massive growth of computer usage has led to an increase in the related security concerns. Malware, such as Viruses, Worms, and Trojans, have become a major issue due to the serious damages they cause. Since the first malware emerged, there has been a continuous battle between security researchers and malware writers, where the latter are constantly trying to evade detection by adopting new functionalities and malicious techniques. This thesis focuses on addressing some of the concerns and challenges encountered when detecting malware, based on their behavioural features observed; for each identified challenge, an approach that addresses the problem is proposed and evaluated. Firstly, the thesis provides an in-depth analysis of the underlying causes of malware misclassification when using machine learning-based malware detectors. Such causes need to be determined, so that the right mitigation can be adopted. The analysis shows that the misclassification is mostly due to changes in several malware variants without the family membership or the year of discovery being a factor. In addition, the thesis proposes a probabilistic approach for optimising the scanning performance of Forensic Virtual Machines (FVMs); which are cloud-based lightweight scanners that perform distributed monitoring of the cloud’s Virtual Machines (VMs). Finally, a market-inspired prioritisation approach is proposed to balance the trade-off between the consumption of VMs’ resources and accuracy when detecting malware on the cloud’s VMs using Virtual Machine Introspection-based lightweight monitoring approaches (e.g. FVMs). The thesis concludes by highlighting future work and new directions that have emerged from the work presented.

[发布机构] University:University of Birmingham;Department:School of Computer Science

[关键词] T Technology;TK Electrical engineering. Electronics Nuclear engineering