A Comparative Analysis of the Effectiveness and Efficiency of Manual and Automated Testing in Software Quality Assurance
DOI:
https://doi.org/10.15294/rji.v4i2.64137Keywords:
Automated Testing, Manual Testing, Software Quality Assurance, Bottleneck, Escaped DefectsAbstract
Purpose: This study aims to analyze the effectiveness, bottlenecks, and reliability of automated testing and manual testing, as well as to compare the two approaches in detecting bugs and supporting software product quality. Accelerating release cycles can be counterproductive if functional stability is ignored; thus, evaluating these two quality assurance methods is crucial to finding an optimal balance in software development.
Methods/Study design/approach: The study employed a quantitative approach through a survey of 105 respondents, comprising QA Engineers, software developers, and academics, using a five-level Likert scale questionnaire. The collected data were rigorously analyzed using descriptive statistics, the Wilcoxon Signed-Rank Test, and the Friedman Test to objectively evaluate the performance, reliability, and operational bottlenecks of both testing methods.
Result/Findings: The results demonstrate that automated testing is highly effective for regression testing and repetitive execution, while manual testing remains significantly effective for UI/UX evaluation and complex dynamic cases. Both methods face distinct challenges. Automated testing is bottlenecked by script maintenance efforts and flaky tests, whereas manual testing is hindered by lengthy execution times and human error. Automated testing was found to be significantly more accurate in detecting recurring functional bugs. Furthermore, an integrated combination of both automated and manual testing was proven to substantially reduce escaped defects.
Novelty/Originality/Value: This research provides empirical evidence confirming that despite the strong push for automation in modern Agile and DevOps environments, neither method can completely replace the other. Both testing approaches are most effective when utilized complementarily according to their intrinsic characteristics. This synergy is essential to prevent bottlenecks in the software release cycle and ensure superior product quality.
References
[1] D. Kato, “Quality Control Methods Using Quality Characteristics in Development and Operations,” Journal of Software Quality Control, pp. 232–243, 2024.
[2] G. Kumari, M. K. Shukla, N. Kumar, R. Kumar, M. S. Adhikari, and S. Singh, “Predicting Test Automation Failures using Machine Learning for Proactive Quality Engineering,” in 6th International Conference on Inventive Computation and Information Technologies, ICICIT 2026 - Proceedings, Institute of Electrical and Electronics Engineers Inc., 2026, pp. 1199–1203. doi: 10.1109/ICICIT69063.2026.11634232.
[3] A. N. Pribadi, S. I. Nuraini, and R. Anggitaningsih, “Peran Quality Assurance Dalam Upaya Penginputan Data Kualitas Benih,” Jurnal Penelitian Nusantara, vol. 1, no. 2, pp. 189–196, 2025.
[4] S. Reine De Reanzi and P. Ranjit Jeba Thangaiah, “A survey on software test automation return on investment, in organizations predominantly from Bengaluru, India,” International Journal of Engineering Business Management, vol. 13, 2021, doi: 10.1177/18479790211062044.
[5] A. R. Rambe and H. Prihantoro, “Pengujian Otomatis Aplikasi Mobile dengan Teknik Black-box Menggunakan Appium (Studi Kasus: Pengembangan Aplikasi Jala Mobile),” Jurnal Automata, vol. 3, no. 2, pp. 1–6, 2022.
[6] C. Zhai, H. Yu, and Z. Liang, “Design and Implementation of Automated Testing for Medical Web Platform Based on Cypress,” China Medical Devices, vol. 39, no. 3, pp. 14–19, 2024, doi: 10.3969/j.issn.1674-1633.2024.03.003.
[7] J. Itkonen, M. V Mäntylä, and C. Lassenius, “The role of the tester’s knowledge in exploratory software testing,” IEEE Transactions on Software Engineering, vol. 39, no. 5, pp. 707–724, 2013, doi: 10.1109/TSE.2012.55.
[8] A. Lama, “A Comparative Study of Manual and Automated Software Testing Strategies in Agile and DevOps Environments,” International Journal of Computer Application, vol. 15, no. 4, pp. 64–68, 2025, doi: 10.5281/zenodo.15844142.
[9] K. Ekasakti and D. N. Pratomo, “Analisis Evaluasi Kinerja Pengujian Manual Dan Otomatis Pada Sistem Informasi Properti (Propertio.id),” Journal of Internet and Software Engineering, vol. 6, no. 2, pp. 88–95, 2025, doi: 10.22146/jise.v6i2.16934.
[10] J. Yu, “Design and Application on Agile Software Exploratory Testing Model,” in Proceedings of 2018 2nd IEEE Advanced Information Management, Communicates, Electronic and Automation Control Conference, IMCEC 2018, B. Xu, Ed., Institute of Electrical and Electronics Engineers Inc., 2018, pp. 2082–2088. doi: 10.1109/IMCEC.2018.8469368.
[11] A. Chatterjee, A. Bala, M. Shah, and A. H. Nagappa, “CTAF: Centralized Test Automation Framework for multiple remote devices using XMPP,” in INDICON 2018 - 15th IEEE India Council International Conference, Institute of Electrical and Electronics Engineers Inc., 2018. doi: 10.1109/INDICON45594.2018.8987182.
[12] F. D. Wicaksono and S. Rani, “Rancang Bangun Automation Test Journey pada E-Commerce (Studi Kasus: Marketplace PT. Tokopedia),” AUTOMATA, vol. 3, no. 2, Aug. 2022. [Online]. Available: https://journal.uii.ac.id/AUTOMATA/article/view/24157
[13] B. Manchuri, “Energy-Efficient Test Automation in Banking: Towards Sustainable CI/CD Pipelines,” in Lecture Notes in Networks and Systems, J. C. Bansal, P. Jamwal, and S. Hussain, Eds., Springer Science and Business Media Deutschland GmbH, 2026, pp. 225–242. doi: 10.1007/978-3-032-22914-4_18.
[14] B. S. Ahmed, A. Gargantini, and M. Bures, “An Automated Testing Framework for Smart TV apps Based on Model Separation,” in Proceedings - 2020 IEEE 13th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2020, Institute of Electrical and Electronics Engineers Inc., 2020, pp. 62–73. doi: 10.1109/ICSTW50294.2020.00026.
[15] C. Paduraru, M. Cernat, and A.-N. Staicu, “A Unified Framework for Automated Testing of Robotic Process Automation Workflows Using Symbolic and Concolic Analysis †,” Machines, vol. 13, no. 6, 2025, doi: 10.3390/machines13060504.
[16] P. Dai, X. Ma, Y. Zhao, and Y. Gong, “Heterogeneous Graph Transformer with Multi-View Representation Learning for Flaky Test Detection,” Computers, vol. 15, no. 6, 2026, doi: 10.3390/computers15060372.
[17] M. Elgazzar, E. Hossny, and F. A. Omara, “A survey of Detecting Flakiness in Automated Test Regression Suite,” in 21st International Learning and Technology Conference: Reality and Science Fiction in Education, L and T 2024, A. Sarirete, E.-A. M. M. F, P. M. Elkafrawy, Z. Balfagih, and T. Brahimi, Eds., Institute of Electrical and Electronics Engineers Inc., 2024, pp. 330–336. doi: 10.1109/LT60077.2024.10469624.
[18] M. Gruber and G. Fraser, “A Survey on How Test Flakiness Affects Developers and What Support They Need To Address It,” in Proceedings - 2022 IEEE 15th International Conference on Software Testing, Verification and Validation, ICST 2022, 2022, pp. 82–92. doi: 10.1109/ICST53961.2022.00020.
[19] O. Parry, G. M. Kapfhammer, M. Hilton, and P. McMinn, “A survey of flaky tests,” ACM Transactions on Software Engineering and Methodology, vol. 31, no. 1, 2021, doi: 10.1145/3476105.
[20] L. Ditasari and T. Raharjo, “Generative AI-Based Software Testing Implementation Model in Agile Development Methodology: Systematic Literature Review,” Journal of Software Engineering, vol. 8, no. 3, pp. 1530–1544, 2026.
[21] A. Sezgin, G. Ozkan, and E. Cosgun, “Leveraging Large Language Models in Software Testing: A Review of Applications and Challenges,” in ISDFS 2025 - 13th International Symposium on Digital Forensics and Security, Institute of Electrical and Electronics Engineers Inc., 2025. doi: 10.1109/ISDFS65363.2025.11011986.
[22] U. K. A. Sethupathy, “DESIGNING A LEAN-AGILE & DEVOPS MATURITY MODEL FOR CONTINUOUS IMPROVEMENT,” Social Science Forum, vol. 8, no. 7, pp. 86–93, 2024, doi: 10.10118/SSFJ/2408710.
[23] M. Lu, T. Cui, Z. Huang, H. Zhao, T. Li, and K. Wang, “A Systematic Review of Questionnaire-Based Quantitative Research on MOOCs,” The International Review of Research in Open and Distributed Learning, vol. 22, no. 2, pp. 285–313, 2021, doi: 10.19173/irrodl.v22i2.5208.
[24] A. Ahmad, O. Leifler, and K. Sandahl, “Empirical analysis of practitioners’ perceptions of test flakiness factors,” Software Testing Verification and Reliability, vol. 31, no. 8, pp. 1–24, 2021, doi: 10.1002/stvr.1791.
[25] I. Dobles, A. Martinez, and C. Quesada-Lopez, “Comparing the effort and effectiveness of automated and manual tests: An industrial case study,” in Iberian Conference on Information Systems and Technologies, CISTI, A. Rocha, I. Pedrosa, M. P. Cota, and R. Goncalves, Eds., IEEE Computer Society, 2019. doi: 10.23919/CISTI.2019.8760848.
[26] R. Haas, D. Elsner, E. Juergens, A. Pretschner, and S. Apel, “How can manual testing processes be optimized? developer survey, optimization guidelines, and case studies,” in ESEC/FSE 2021 - Proceedings of the 29th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2021, pp. 1281–1291. doi: 10.1145/3468264.3473922.
[27] B. Wang, L. Zhu, D. Sun, W. Wang, S. Song, and S. Peng, “Towards an Automated Testing Framework for IoT Devices,” in Journal of Physics: Conference Series, Institute of Physics, 2023. doi: 10.1088/1742-6596/2493/1/012023.
[28] Y. Wang, M. V Mäntylä, Z. Liu, J. Markkula, and P. Raulamo-jurvanen, “Improving test automation maturity: A multivocal literature review,” Software Testing Verification and Reliability, vol. 32, no. 3, pp. 1–33, 2022, doi: 10.1002/stvr.1804.
[29] D. L. Lima, R. De Souza Santos, G. P. Garcia, S. S. Da Silva, C. Franca, and L. F. Capretz, “Software Testing and Code Refactoring: A Survey with Practitioners,” in Proceedings - 2023 IEEE International Conference on Software Maintenance and Evolution, ICSME 2023, 2023, pp. 500–507. doi: 10.1109/ICSME58846.2023.00064.
[30] Q. Cui, J. Wang, G. Yang, M. Xie, Q. Wang, and M. Li, “Who Should Be Selected to Perform a Task in Crowdsourced Testing?,” in Proceedings - International Computer Software and Applications Conference, S. Reisman, S. I. Ahamed, C. Demartini, T. Conte, L. Liu, W. Claycomb, M. Nakamura, E. Tovar, S. Cimato, L. C.-H., H. Takakura, Y. J.-J., T. Akiyama, Z. Zhang, and K. Hasan, Eds., IEEE Computer Society, 2017, pp. 75–84. doi: 10.1109/COMPSAC.2017.265.
[31] A. Rainer and C. Wohlin, “Recruiting credible participants for field studies in software engineering research,” Inf. Softw. Technol., vol. 151, p. 107002, 2022, doi: 10.1016/j.infsof.2022.107002.
[32] S. O. Barraood, H. Mohd, and F. Baharom, “An initial investigation of the effect of quality factors on Agile test case quality through experts’ review,” Cogent Eng., vol. 9, no. 1, 2022, doi: 10.1080/23311916.2022.2082121.
[33] B. Jiang, “The Application of Vue.js Framework Technology in Multidomain Automated Testing Systems,” Automatic Control and Computer Sciences, vol. 59, no. 4, pp. 455–466, 2025, doi: 10.3103/S0146411625700610.
[34] R. Almeida, S. Nogueira, and A. Sampaio, “Combining sequential feature test cases to generate sound tests for concurrent features,” Sci. Comput. Program., vol. 250, 2026, doi: 10.1016/j.scico.2025.103414.
[35] N. Mani and S. Attaranasl, “Enhancing Adaptive Test Healing with Graph Neural Networks for Dependency-Aware Decision Making,” in Proceedings - 2025 IEEE International Conference on Artificial Intelligence Testing, AITest 2025, Institute of Electrical and Electronics Engineers Inc., 2025, pp. 126–133. doi: 10.1109/AITest66680.2025.00023.
[36] N. Mani and S. Attaranasl, “Adaptive Test Healing using LLM/GPT and Reinforcement Learning,” in 2025 IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2025, A. R. Fasolino, S. Panichella, A. Aleti, and A. Mesbah, Eds., Institute of Electrical and Electronics Engineers Inc., 2025, pp. 9–16. doi: 10.1109/ICSTW64639.2025.10962516.
[37] Z. Khaliq, S. U. Farooq, and D. A. Khan, “A deep learning-based automated framework for functional User Interface testing,” Inf. Softw. Technol., vol. 150, 2022, doi: 10.1016/j.infsof.2022.106969.
[38] W. Lu, A. Senchenko, A. Sayle, A. Hindle, and C.-P. Bezemer, “How Far Can VLMs Go for Visual Bug Detection? Studying 19,738 Keyframes from 41 Hours of Gameplay Videos,” in FSE Companion 2026 - Proceedings of the 34th ACM International Conference on the Foundations of Software Engineering, S. H. Tan and F. Khomh, Eds., Association for Computing Machinery, Inc, 2026, pp. 345–350. doi: 10.1145/3803437.3805208.
[39] A. Myrria et al., “Leveraging Template Matching for Efficient Android Screenshot Visual Inspection,” in Digest of Technical Papers - IEEE International Conference on Consumer Electronics, Institute of Electrical and Electronics Engineers Inc., 2026. doi: 10.1109/ICCE67443.2026.11449809.
[40] A. Perera, “Using Defect Prediction to Improve the Bug Detection Capability of Search-Based Software Testing,” in Proceedings - 2020 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020, Institute of Electrical and Electronics Engineers Inc., 2020, pp. 1170–1174. doi: 10.1145/3324884.3415286.
[41] S. Wongkampoo and S. Kiattisin, “Atom-Task Precondition Technique to Optimize Large Scale GUI Testing Time based on Parallel Scheduling Algorithm,” in ICSEC 2017 - 21st International Computer Science and Engineering Conference 2017, Proceeding, Institute of Electrical and Electronics Engineers Inc., 2018, pp. 229–232. doi: 10.1109/ICSEC.2017.8443913.
[42] X. Wu et al., “Widget Detection-based Testing for Industrial Mobile Games,” in Proceedings - International Conference on Software Engineering, IEEE Computer Society, 2023, pp. 173–184. doi: 10.1109/ICSE-SEIP58684.2023.00021.
[43] L. Borzacchiello, E. Coppa, and C. Demetrescu, “FUZZOLIC: Mixing fuzzing and concolic execution,” Comput. Secur., vol. 108, 2021, doi: 10.1016/j.cose.2021.102368.
[44] V. Chawla, “Enhancing Security and Performance in PyTorch: A Hybrid Fuzzing Approach,” in Proceedings of the 1st International Symposium on Parallel Computing and Distributed Systems, PCDS 2024, C. W. Tan and T. H. Teo, Eds., Institute of Electrical and Electronics Engineers Inc., 2024. doi: 10.1109/PCDS61776.2024.10743720.
[45] D. Parygina, A. Vishnyakov, and A. Fedotov, “Strong Optimistic Solving for Dynamic Symbolic Execution,” in Proceedings - 2022 Ivannikov Memorial Workshop, IVMEM 2022, Institute of Electrical and Electronics Engineers Inc., 2022, pp. 43–53. doi: 10.1109/IVMEM57067.2022.9983965.
[46] M. Staron and S. Abrahão, “Hybrid Classical-AI Systems for Software Testing and Bug Fixing,” IEEE Softw., vol. 42, no. 6, pp. 106–110, 2025, doi: 10.1109/MS.2025.3597698.
[47] J. Yang and W. K. Chan, “Dynamic Testing Against Hidden Concurrency Bugs through Abstraction and Projection,” in Proceedings - International Computer Software and Applications Conference, S. Reisman, S. I. Ahamed, L. Liu, D. Milojicic, W. Claycomb, M. Matskin, H. Sato, M. Nakamura, S. Cimato, C. H. Lung, Z. Zhang, and Z. Zhang, Eds., IEEE Computer Society, 2016, pp. 600–601. doi: 10.1109/COMPSAC.2016.102.
