Real-Time Multi-Class DoS Attack Detection on Proxmox VMs UsingLightGBM with MikroTik Integration

Abstract

Purpose: The adoption of virtualization increases the dependence on service availability, making Denial of Service
(DoS) attacks a serious threat, while rule-based detection is poorly adaptive to evolving attacks. Many previous studies also rely on outdated public datasets, are evaluated offline, rarely measure inference latency, and lack automatic mitigation. This study aims to build a real-time multi-class DoS detection system on Proxmox virtual machines using LightGBM integrated with MikroTik.
Methods: A controlled testbed based on Proxmox and MikroTik was built to generate normal and attack traffic. The dataset was collected from the real infrastructure at a one-second granularity and labeled into six classes, namely the normal condition and five DoS attacks. LightGBM was proposed as the detection model, while XGBoost, Random Forest, Decision Tree, and SVM served as baselines, compared using a temporal holdout to prevent data leakage, with SMOTE applied only to the training data.
Findings: LightGBM was selected as the best model with an accuracy of 96.22%, a macro F1-score of 96.25%, and an inference latency of 1.369 ms. The four flooding attacks were detected almost perfectly, whereas Slowloris was the hardest class because it resembles normal traffic. Its PR-AUC dropped to 0.9271, and the system performed automatic mitigation at a median latency of 56.2 ms.
Originality: This study integrates lightweight real-time detection with automatic firewall-based mitigation in a closed loop on real infrastructure, emphasizing the balance between accuracy and efficiency rather than the highest accuracy alone. Future work can extend it to distributed (DDoS) attacks.