AI-Driven Malware Behaviour Classification and Detection Systems

Abstract

The contemporary cybersecurity landscape is characterized by an incessant arms race between malicious actors designing increasingly sophisticated malware and defenders seeking to detect and mitigate these threats effectively. Traditional signature-based antivirus solutions, which rely on known patterns and static characteristics, are rendered largely impotent against novel, polymorphic, and metamorphic malware that can adapt their code to evade detection. In response, the industry has witnessed a paradigm shift toward behavior-based detection systems empowered by artificial intelligence (AI) and machine learning (ML). This manuscript delves into an AI-driven framework for dynamic malware behavior classification and real-time detection, detailing the processes of comprehensive feature extraction, rigorous statistical analysis, classifier training, and simulated deployment in enterprise environments.

We curated a diverse dataset comprising 5,000 benign and 5,000 malicious Windows PE samples, executing each in a controlled sandbox environment to capture API call sequences, network traffic metrics, file system interactions, and registry modifications over a five-minute runtime.

Features were aggregated into temporal and frequency-based descriptors, yielding a feature vector of 120 attributes per sample. Dimensionality reduction via Principal Component Analysis (PCA) preceded supervised learning using Random Forest (RF), Support Vector Machine (SVM), and Deep Neural Network (DNN) models. The RF classifier demonstrated superior performance with 97.8% accuracy, 96.5% precision, and 98.1% recall under four-fold cross-validation. Statistical testing (two-sample t-tests, Mann–Whitney U) corroborated the discriminative power of key dynamic features (p < 0.001).

A simulation testbed, implemented in NS-3, emulated an enterprise network of 200 hosts engaging in regular productivity traffic interspersed with stealth and burst malware injection scenarios. The RF-based detection pipeline, deployed as a RESTful microservice, achieved an average detection latency of 120 ms, maintained CPU utilization under 65% during peak loads, and sustained a 99% detection rate with zero false positives in stealth mode. The proposed system thus offers a robust, adaptive solution for cybersecurity operations centers (SOCs) and Security Information and Event Management (SIEM) platforms, capable of countering emerging threats with minimal human intervention.