Robustness of Bayesian-Optimized Random Forest Against Feature-Space Evasion Attacks for Spam Detection
DOI:
https://doi.org/10.15294/rji.v4i2.62732Keywords:
Email spam detection, Random Forest, adversarial robustnes, feature-space evasion attacksAbstract
Abstract. Email spam filters may perform well on unmodified messages yet remain vulnerable to deliberate feature manipulation. Although hyperparameter optimization can improve clean-data performance, such gains do not establish robustness under test-time perturbations. A paired, leakage-controlled evaluation is therefore needed to assess both properties under identical conditions.
Purpose: This study compared the clean classification performance and feature-space robustness of a library-default Random Forest (RF-default) and a Bayesian-optimized Random Forest (BO-RF) on Spambase.
Methods/Study design/approach: Exact duplicates were removed, and predictor-identical records were grouped to prevent cross-fold leakage. Evaluation used two repetitions of five-fold nested stratified group-aware cross-validation with shared outer folds. Bayesian Optimization maximized inner-validation spam-class F1. Frozen outer-test folds were assessed on clean inputs, class-center-directed perturbations at α = 0.05, 0.10, 0.20, and 0.40, and 30 magnitude-matched random directions. Retained inference assessed within-model clean-versus-directed and directed-versus-random contrasts in F1, Recall, and model-specific attack success rate (ASR).
Result/Findings: BO-RF yielded slightly higher means across all four clean metrics; clean F1 was 93.65 ± 0.88% versus 93.53 ± 0.98% for RF-default. Bayesian Optimization selected the effective default configuration in nine of ten outer folds. At directed α = 0.40, F1 fell to 85.20% for RF-default and 85.31% for BO-RF, while Recall fell to 77.94% and 78.03%, respectively; directed ASR was approximately 15.5% for both models. All 48 retained within-model contrasts met Holm-adjusted thresholds, indicating degradation relative to clean inputs and a stronger effect of directed than magnitude-matched random perturbations. No direct between-model inferential family was retained, so robustness comparisons remain descriptive.
Novelty/Originality/Value: Under the evaluated Spambase feature-space setting, clean-F1 Bayesian Optimization yielded only marginal descriptive gains and did not demonstrate a clear robustness advantage over the effective RF-default configuration. The findings show that clean-metric optimization should not be treated as a proxy for evasion robustness.
References
[1] F. Jáñez-Martino, R. Alaiz-Rodríguez, V. González-Castro, E. Fidalgo, and E. Alegre, “A review of spam email detection: analysis of spammer strategies and the dataset shift problem,” Artif. Intell. Rev., vol. 56, no. 2, pp. 1145–1173, Feb. 2023, doi: 10.1007/s10462-022-10195-4.
[2] B. Biggio et al., “Evasion Attacks against Machine Learning at Test Time,” in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 8190 LNAI, no. PART 3, Springer, Berlin, Heidelberg, 2013, pp. 387–402. doi: 10.1007/978-3-642-40994-3_25.
[3] B. Biggio and F. Roli, “Wild patterns: Ten years after the rise of adversarial machine learning,” Pattern Recognit., vol. 84, pp. 317–331, Dec. 2018, doi: 10.1016/j.patcog.2018.07.023.
[4] B. Biggio, G. Fumera, and F. Roli, “Security Evaluation of Pattern Classifiers under Attack,” IEEE Trans. Knowl. Data Eng., vol. 26, no. 4, pp. 984–996, Apr. 2014, doi: 10.1109/TKDE.2013.57.
[5] L. Breiman, “Random Forests,” Mach. Learn., vol. 45, no. 1, pp. 5–32, Oct. 2001, doi: 10.1023/A:1010933404324.
[6] M. A. Bouke, O. I. Alramli, and A. Abdullah, “XAIRF-WFP: a novel XAI-based random forest classifier for advanced email spam detection,” Int. J. Inf. Secur., vol. 24, no. 1, p. 5, Feb. 2025, doi: 10.1007/s10207-024-00920-1.
[7] T. O. Omotehinwa and D. O. Oyewola, “Hyperparameter Optimization of Ensemble Models for Spam Email Detection,” Applied Sciences, vol. 13, no. 3, p. 1971, Feb. 2023, doi: 10.3390/app13031971.
[8] P. Probst, M. N. Wright, and A. Boulesteix, “Hyperparameters and tuning strategies for random forest,” WIREs Data Mining and Knowledge Discovery, vol. 9, no. 3, p. e1301, May 2019, doi: 10.1002/widm.1301.
[9] J. Snoek, H. Larochelle, and R. Adams, “Practical Bayesian Optimization of Machine Learning Algorithms,” in Advances in Neural Information Processing Systems, F. Pereira, C. J. Burges, L. Bottou, and K. Weinberger, Eds., Curran Associates, Inc., 2012. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2012/file/05311655a15b75fab86956663e1819cd-Paper.pdf
[10] X. Wang, Y. Jin, S. Schmitt, and M. Olhofer, “Recent Advances in Bayesian Optimization,” ACM Comput. Surv., vol. 55, no. 13s, pp. 1–36, Dec. 2023, doi: 10.1145/3582078.
[11] S. Goyal, S. Doddapaneni, M. M. Khapra, and B. Ravindran, “A Survey of Adversarial Defenses and Robustness in NLP,” ACM Comput. Surv., vol. 55, no. 14s, pp. 1–39, Dec. 2023, doi: 10.1145/3593042.
[12] Q. Cheng, A. Xu, X. Li, and L. Ding, “Adversarial Email Generation against Spam Detection Models through Feature Perturbation,” in 2022 IEEE International Conference on Assured Autonomy (ICAA), IEEE, Mar. 2022, pp. 83–92. doi: 10.1109/ICAA52185.2022.00019.
[13] E. Hotoğlu, S. Sen, and B. Can, “A comprehensive analysis of adversarial attacks against spam filters,” Comput. Secur., in press, Art. no. 105066, available online Jul. 23, 2026, doi: 10.1016/j.cose.2026.105066.
[14] F. Yang, Z. Chen, and A. Gangopadhyay, “Using Randomness to Improve Robustness of Tree-Based Models Against Evasion Attacks,” IEEE Trans. Knowl. Data Eng., vol. 34, no. 2, pp. 969–982, Feb. 2022, doi: 10.1109/TKDE.2020.2987299.
[15] F. Pierazzi, F. Pendlebury, J. Cortellazzi, and L. Cavallaro, “Intriguing Properties of Adversarial ML Attacks in the Problem Space,” in 2020 IEEE Symposium on Security and Privacy (SP), IEEE, May 2020, pp. 1332–1349. doi: 10.1109/SP40000.2020.00073.
[16] K. Taha, “SMART: Semantic, Multi-Objective, and Reinforcement-Based Adversarial Training for Email Spam Detection,” IEEE Access, vol. 13, pp. 112749–112764, 2025, doi: 10.1109/ACCESS.2025.3581131.
[17] M. Hopkins, E. Reeber, G. Forman, and J. Suermondt, “Spambase,” UCI Machine Learning Repository. Accessed: May 29, 2026. [Online]. Available: https://doi.org/10.24432/C53G6X
[18] F. Pedregosa et al., “Scikit-learn: Machine Learning in Python,” Journal of Machine Learning Research, vol. 12, no. 85, pp. 2825–2830, 2011, [Online]. Available: http://jmlr.org/papers/v12/pedregosa11a.html
[19] F. Nogueira, “Bayesian Optimization: Open source constrained global optimization tool for Python,” 2014, Accessed: Jun. 19, 2026. [Online]. Available: https://github.com/bayesian-optimization/BayesianOptimization
[20] N. Srinivas, A. Krause, S. M. Kakade, and M. W. Seeger, “Information-Theoretic Regret Bounds for Gaussian Process Optimization in the Bandit Setting,” IEEE Trans. Inf. Theory, vol. 58, no. 5, pp. 3250–3265, May 2012, doi: 10.1109/TIT.2011.2182033.
[21] G. C. Cawley and N. L. C. Talbot, “On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation,” Journal of Machine Learning Research, vol. 11, pp. 2079–2107, 2010.
[22] M. Sokolova and G. Lapalme, “A systematic analysis of performance measures for classification tasks,” Inf. Process. Manag., vol. 45, no. 4, pp. 427–437, Jul. 2009, doi: 10.1016/j.ipm.2009.03.002.
[23] R. R. Bouckaert and E. Frank, “Evaluating the Replicability of Significance Tests for Comparing Learning Algorithms,” 2004, pp. 3–12. doi: 10.1007/978-3-540-24775-3_3.
[24] C. Nadeau and Y. Bengio, “Inference for the Generalization Error,” Mach. Learn., vol. 52, no. 3, pp. 239–281, Sep. 2003, doi: 10.1023/A:1024068626366.
[25] S. Holm, “A Simple Sequentially Rejective Multiple Test Procedure,” Scandinavian Journal of Statistics, vol. 6, pp. 65–70, 1979.
