Enhancing Malware Classification with Machine Learning: A Comparative Analysis of API Sequence-Based Techniques

Abstract

Malware samples have been widely used for unauthorized objectives. New varieties of malware are discovered daily. The application of machine learning (ML) for system and network security is one of the key areas of current study due to its effectiveness and rapid advancement over the past decade. In our research work, we explored the implementation of ML in malware classification and analysis by enabling dynamic and adaptive threat recognition. This research introduces a pioneering methodology for categorizing malware samples into their respective families by leveraging ML techniques based on application programming interface (API) sequences and arguments. The experiment encompassed nine distinct malware families, along with benign samples, collected for analysis. Several ML models are used for feature learning and classification. These include convolutional neural network (CNN), naive bayes, random forest, and XGBoost. The outcomes of the experiments show that there are significant disparities in the performance of the models. Specifically, SGD attained an accuracy of 79.90%, CNN achieved 72.53% accuracy, Naive Bayes demonstrated 8.7% accuracy, Random Forest yielded 92.37% accuracy, while XGBoost emerged as the front runner with an accuracy of 98.87%. These findings show the effectiveness of employing API sequences and arguments for malware classification in XGBoost. XGBoost accurately categorizes malicious samples into their respective families.