Malware Family Classification from Behavioral API Multigraphs Using Edge-Aware Graph Neural Networks

Abstract

Malware family classification from dynamic analysis traces remains challenging because most existing approaches represent sandbox behavior as a flat API-call sequence or an aggregated feature vector, limiting their ability to capture the relationships among API calls. In this work, we present a graph-based framework for malware family classification from dynamic behavioral analysis. Each malware execution report generated by the CAPEv2 sandbox is modeled as a directed behavioral API multigraph, where nodes represent unique API names and edges preserve the corresponding call arguments, semantic category, and execution status. The graphs are classified using an edge-aware graph neural network with edge-aware message passing, graph attention, residual connections, and dual mean–max pooling. Focal loss, class-balanced weighting, and post-hoc threshold calibration are used to address severe class imbalance. Evaluated on the public Avast-CTU CAPEv2 dataset containing 48,976 samples from ten malware families under a temporal evaluation protocol, the proposed model achieves a macro-F1 score of 94.04% as a single model and 95.25% as a five-model ensemble.The proposed model significantly outperforms four non-graph baselines and achieves the highest macro-F1 among two existing graph-based malware classifiers evaluated under a common experimental protocol. A controlled comparison across five graph convolution operators and a component ablation study show that performance depends more on the overall framework than on the specific graph convolution operator, and that hand-crafted graph-level statistics are largely unnecessary. The interpretability analysis further shows that the remaining classification errors mainly arise from behavioral overlap among credential-stealing malware families.