Virtual Machine Introspection Based Malware Detection Approach at Hypervisor for Virtualized Cloud Computing Environment

Abstract

Cloud computing enabled by virtualization technology exhibits a revolutionary change in information technology infrastructure. The hypervisor is a pillar of virtualization and it allows to abstract the host or bare hardware resources to the Virtual Machines (VMs) which are running on the virtualized environment. As the VMs are easily available for rent from the Cloud Service Provider (CSP) that are a prime target for malignant cloud user or an adversary to launch the attacks and to execute the sophisticated malware by exploiting the identi ed vulnerability present in it. In addition, the proliferation of unknown malware exposes the limitations of traditional and VM-based anti-malware defensive solutions. These motivated the development of secure hypervisor or Virtual Machine Monitor (VMM) based solutions. The Virtual Machine Introspection (VMI) has emerged as a ne-grained out-of-VM security solution to detect the malware by introspecting and reconstructing the volatile memory state of the live guest Operating System (OS) by functioning at VMM. However, VMM-based introspection solutions present a number of limitations, including the well-known semantic gap issue. In this work, as a rst proposed work (methodology) we study the limitation of existing host-based security solution. To address this issue, we proposed Virtual Machine and hypervisor based Intrusion Detection and Prevention System (VMIDPS) for virtualized environment to ensure the robust state of the VM by detecting and analyzing the rootkits as well as other attacks on live monitored guest OS. The VMIDPS leverages cross-view based technique for detection and identi cation of intrusion at VM. The experimental results showed that the VMIDPS successfully detected the Windows based rootkits, and Denial of Service (DoS) attack on Monitored VMs. However, the main limitation of this approach is that it uses an agent-based solution on each of the individual Monitored VM to obtain the run state of the guest OS. In the second proposed work (methodology), we study the limitation of the prior VMI technique that is not intelligent enough to read precisely the manipulated semantic information on their reconstructed high-level semantic view of the live guest OS at VMM. To e ectively address this issue, we proposed VMI-based real-time imalware detection system called Automated-Internal-External (A-IntExt) system. It seamlessly introspects the untrustworthy Windows guest OS internal semantic view (i.e., processes). Further, it checks the detected, hidden as well as running processes (not hidden) as benign or malicious. The prime component of the A-IntExt system is the Intelligent Cross-View Analyzer (ICVA) that leverages the novel Time Interval Threshold (TIT) technique for detecting the hidden-state information from the internally and externally gathered run state information of the Monitored VM. Experimental results showed that, we can e ectively detect and manually analyze the stealthy hidden activity of the malware and rootkits, including measurement with Windows benchmark programs. In the third proposed work (methodology), we have further extended the A-IntExt system as an advanced VMM-based guest-assisted Automated Multi-level Malware Detection System (AMMDS) that leverages both VMI and Memory Forensic Analysis (MFA) techniques to predict early symptoms of malware execution by detecting stealthy hidden processes on a live guest OS. The AMMDS generalize the cyber physical system application that is functioning at introspected guest OS. More speci cally, the AMMDS detects and classi es the actual running malicious executables from the semantically reconstructed executables (i.e., .exe) the process view of the guest OS. The two sub-components of the AMMDS are: Online Malware Detector (OMD) and O ine Malware Classi er (OFMC). The OMD recognizes whether the running processes are benign or malicious using its Local Malware Signature Database (LMSD) and OMS. The OFMC classi es unknown malware by adopting machine learning techniques at hypervisor. The AMMDS has been evaluated by executing large real-world malware and benign executables on to the live guest OSs. The evaluation results achieved full detection accuracy in classifying unknown malware with a considerable performance overhead. In the fourth proposed work (methodology), we have systematically evaluated other shortcomings of our proposed A-IntExt system and AMMDS. In this work, we further extended the A-IntExt system by implementing Hybrid Feature (HF) selection technique that uses representative instances of other individual feature selection techniques of the corresponding feature set that were extracted from the detected hidden and dubious executables of infected memory dumps of the introspected guest OSs. iiFurther, the proposed approach has been validated with other public benchmarked datasets at VMM. The AMMDS also performs o ine detection of malware, however, it fails to address the over-fitting issue that plagues many machine learning techniques. In this work, we precisely address the over-fitting issue by dividing both generated dataset (VMM level) and benchmarked datasets as training, testing and validation sets. The evaluation results showed that proposed approach is pro cient in detecting unknown malware with high detection accuracy on both generated and benchmarked datasets. In the fth work, the execution time of the MFA tools such as Volatility and Rekall is measured and compared for the di erent RAM dump sizes. The motivation behind this works is that RAM dump capture time and its analysis time in real time are highly crucial if an IDS depends on data supplied by the MFA tool or VMI tool. Furthermore, analysis of malware based on the infected memory dump is also a primary for an IDS. In this context, the evaluation conducted on memory dumps of both Linux and Windows VMs that are captured using open source VMI tool called LibVMI.