Sentiment Analysis of Generative AI Application Reviews Using Random Forest and SVM with SMOTE-Based Class Imbalance Handling
DOI:
https://doi.org/10.15294/rji.v4i2.60530Keywords:
Generative AI, Random Forest, Sentiment Analysis, SMOTE, Support Vector MachineAbstract
Abstract. The rapid adoption of Generative Artificial Intelligence (Gen-AI) has generated extensive user reviews, offering valuable insights into user perceptions. This study aims to classify review sentiment using Random Forest and Support Vector Machine (SVM), evaluate the impact of SMOTE on class imbalance, and analyze review themes. This study analyzed 50,000 reviews from five popular Gen-AI applications: ChatGPT, Microsoft Copilot, Google Gemini, Perplexity, and Claude. The models were evaluated using accuracy, weighted precision, weighted recall, weighted F1-score, and confusion matrix. The results showed that SVM without SMOTE achieved the highest overall accuracy of 84% and a weighted F1-score of 0.81. However, its F1-score for the neutral class was only 0.06. Applying SMOTE reduced the overall accuracy of Random Forest from 83% to 81% and SVM from 84% to 76%. Nevertheless, SMOTE increased the neutral-class F1-score from 0.00 to 0.06 for Random Forest and from 0.06 to 0.20 for SVM. These results indicate that SMOTE improved minority-class recognition, particularly for SVM, but introduced a trade-off with overall predictive performance. Ultimately, these findings provide practical insights for improving Gen-AI applications while also serving as a reference for selecting sentiment classification models in imbalanced multiclass sentiment analysis.
Purpose: This study aims to classify the sentiment of user reviews on Gen-AI applications using Random Forest and Support Vector Machine (SVM), evaluate the impact of SMOTE on imbalanced multiclass data, and identify key review themes influencing user perceptions.
Methods/Study design/approach: A dataset of 50,000 user reviews from five popular Gen-AI applications was analyzed using text preprocessing, TF-IDF feature extraction, rating-based sentiment labeling, and SMOTE applied to the training data. Random Forest and SVM were evaluated using accuracy, weighted precision, weighted recall, weighted F1-score, and confusion matrix.
Result/Findings: SVM without SMOTE achieved the best overall performance, with an accuracy of 84% and a weighted F1-score of 0.81. Although SMOTE reduced overall accuracy, it improved the neutral-class F1-score from 0.00 to 0.06 for Random Forest and from 0.06 to 0.20 for SVM, demonstrating better recognition of minority-class samples.
Novelty/Originality/Value: This study provides a comparative evaluation of Random Forest and SVM with and without SMOTE for multiclass sentiment analysis of Gen-AI user reviews. It also integrates review theme analysis to identify factors affecting user perceptions, offering practical guidance for selecting classification models and improving Gen-AI applications.
References
[1] A. Holzinger, K. Keiblinger, P. Holub, K. Zatloukal, and H. Müller, “AI for life : Trends in artificial intelligence for biotechnology,” New Biotechnology, vol. 74, pp. 16–24, 2023, doi: 10.1016/j.nbt.2023.02.001.
[2] P. Ågerfalk, P. J., Conboy, K., Crowston, K., Eriksson Lundström, J. S., Jarvenpaa, S., Ram, S., & Mikalef, “Artificial Intelligence in Information Systems : State of the Art and Research Roadmap,” Communications of the Association for Information Systems, vol. 50, no. 1, pp. 420–438, 2022, doi: 10.17705/1CAIS.05017.
[3] M. F. Shahzad, S. Xu, X. An, and M. Asif, “Are Generative AI Technologies Transforming Education for the 21st Century? Research Trends, Challenges, and Benefits,” Sage Open, vol. 15, no. 3, Art. no. 21582440251368594, 2025, doi: 10.1177/21582440251368594.
[4] M. Trigka and E. Dritsas, “The Evolution of Generative AI: Trends and Applications,” IEEE Access, vol. 13, pp. 98504–98529, 2025, doi: 10.1109/ACCESS.2025.3574660.
[5] S. Bouguettaya, F. Pupo, M. Chen, and G. Fortino, “A Meta-Survey of Generative AI in Education: Trends, Challenges, and Research Directions,” Big Data and Cognitive Computing, vol. 9, no. 9, p. 237, 2025, doi: 10.3390/bdcc9090237.
[6] S. Feuerriegel, J. Hartmann, C. Janiesch, and P. Zschech, “Generative AI,” Business and Information Systems Engineering., vol. 66, no. 1, pp. 111–126, 2024, doi: 10.1007/s12599-023-00834-7.
[7] X. Sun and W. He, “Exploring the Frontier of Generative AI: Evolution, Applications, and Future Trends,” International Journal of Crowd Science., vol. 9, no. 4, pp. 269–277, 2025, doi: 10.26599/IJCS.2024.9100032.
[8] R. Alabduljabbar, “User-centric AI: Evaluating the Usability of Generative AI Applications Through User Reviews on App Stores,” PeerJ Computer Science., vol. 10, Art. no. e2421, 2024, doi: 10.7717/peerj-cs.2421.
[9] T. Chen, P. Samaranayake, X. Cen, M. Qi, and Y.-C. Lan, “The Impact of Online Reviews on Consumers’ Purchasing Decisions: Evidence From an Eye-Tracking Study,” Frontiers in Psychology, vol. 13, Art. no. 865702, 2022, doi: 10.3389/fpsyg.2022.865702.
[10] F. Muttakin, N. Andrika, and Salsabila, “Sentiment Analysis of Shoe Product Reviews on Indonesian E-Commerce Platform Using Lexicon Based and Support Vector Machine,” Jurnal Teknik Informatika (JUTIF), vol. 6, no. 2, pp. 839–854, 2025, doi: 10.52436/1.jutif.2025.6.2.3800.
[11] H. Imaduddin, F. Y. A’la, and Y. S. Nugroho, “Sentiment Analysis in Indonesian Healthcare Applications using IndoBERT Approach,” International Journal of Advanced Computer Science and Applications, vol. 14, no. 8, pp. 113–117, 2023, doi: 10.14569/IJACSA.2023.0140813.
[12] M. D. R. Haque and S. Rubya, “An Overview of Chatbot-Based Mobile Mental Health Apps: Insights From App Description and User Reviews,” JMIR mHealth uHealth, vol. 11, Art. no. e44838, 2023, doi: 10.2196/44838.
[13] J. Sangeetha and U. Kumaran, “Sentiment Analysis of Amazon User Reviews Using a Hybrid Approach,” Measurement: Sensors, vol. 27, Art. no. 100790, 2023, doi: 10.1016/j.measen.2023.100790.
[14] A. Albladi, M. Islam, and C. Seals, “Sentiment Analysis of Twitter Data Using NLP Models: A Comprehensive Review,” IEEE Access, vol. 13, pp. 30444–30468, 2025, doi: 10.1109/ACCESS.2025.3541494.
[15] S. Dahl, M. Bøgsted, T. Sagi, and C. Vesteghem, “Performance of Natural Language Processing for Information Extraction From Electronic Health Records Within Cancer: Systematic Review,” JMIR Medical Informatics, vol. 13, Art. no. e68707, 2025, doi: 10.2196/68707.
[16] F. Aftab et al., “A Comprehensive Survey on Sentiment Analysis Techniques,” International Journal of Technology, vol. 14, no. 6, pp. 1288–1298, 2023, doi: 10.14716/ijtech.v14i6.6632.
[17] R. D. Kurniawan, A. Yohannis, and W. T. Atmojo, “Sentiment Analysis of Getcontact Application Reviews on Google Play Store Using Naive Bayes Algorithm,” Jurnal Teknik Informatika (JUTIF), vol. 6, no. 4, pp. 2848–2858, 2025, doi: 10.52436/1.jutif.2025.6.4.5248.
[18] R. Nurhasanah and V. A. Elyakim, “Analisis Sentimen Ulasan Pengguna Generative AI Menggunakan Naïve Bayes Berbasis Python,” INSOLOGI: Jurnal Sains dan Teknologi, vol. 5, no. 3, pp. 875–890, 2026, doi: 10.55123/insologi.v5i3.8050.
[19] S. H. Imanuddin, K. Adi, and R. Gernowo, “Sentiment Analysis on Satusehat Application Using Support Vector Machine Method,” Journal of Electronics, Electromedical Engineering, and Medical Informatics, vol. 5, no. 3, pp. 143–149, 2023, doi: 10.35882/jeemi.v5i3.304.
[20] M. R. Rahman, A. F. Diansyah, and Hanafi, “Sentiment Analysis on the Shopee Application on Playstore Using the Random Forest Classification Method,” Inform : Jurnal Ilmiah Bidang Teknologi Informasi dan Komunikasi, vol. 9, no. 1, pp. 20–24, 2024, doi: 10.25139/inform.v9i1.5465.
[21] R. R. Pratama and R. R. Suryono, “Performance Comparison of Naive Bayes, Support Vector Machine and Random Forest Algorithms for Apple Vision Pro Sentiment Analysis,” Jurnal Teknik Informatika (JUTIF), vol. 6, no. 1, pp. 31–40, 2025, doi: 10.52436/1.jutif.2025.6.1.4035.
[22] A. Shobil and S. Al-Hagree, “User Sentiment Analysis of Solar Energy Monitoring Apps: Insights From Google Play Reviews,” IEEE Access, vol. 14, pp. 38421–38433, 2026, doi: 10.1109/ACCESS.2026.3670366.
[23] D. Kartika and S. Defit, “Optimizing Naive Bayes for Sentiment Analysis of M-Passport Reviews Using N-Gram and Synthetic Minority Over-sampling Technique,” Jurnal Teknik Informatika (JUTIF), vol. 7, no. 3, pp. 2798–2811, 2026, doi: 10.52436/1.jutif.2026.7.3.5853.
[24] P. P. Putra, M. K. Anam, A. S. Chan, A. Hadi, N. Hendri, and A. Masnur, “Optimizing Sentiment Analysis on Imbalanced Hotel Review Data Using SMOTE and Ensemble Machine Learning Techniques,” Journal of Applied Data Sciences, vol. 6, no. 2, pp. 936–951, 2025, doi: 10.47738/jads.v6i2.618.
[25] Badriyah, T. Chamidy, and Suhartono, “Application of SMOTE in Sentiment Analysis of MyXL User Reviews on Google Play Store,” JISKA (Jurnal Informatika Sunan Kalijaga), vol. 10, no. 1, pp. 74–86, 2025, doi: 10.14421/jiska.2025.10.1.74-86.
[26] J. Kachhia, “The Generative AI Ecosystem: 50K User Reviews 2026,” Kaggle, 2026. https://www.kaggle.com/datasets/jahnavikachhia23/the-generative-ai-ecosystem-50k-user-reviews-2026 (accessed Jun. 25, 2026).
[27] H. Syah and A. Witanti, “Analisis Sentimen Masyarakat Terhadap Vaksinasi COVID-19 pada Media Sosial Twitter Menggunakan Algoritma Support Vector Machine (SVM),” Jurnal Sistem Informasi dan Informatika (SIMIKA), vol. 5, no. 1, pp. 59–67, 2022, doi: 10.47080/SIMIKA.V5I1.1411.
[28] I. Najiyah and M. Rizal, “Analisis Sentimen Media Sosial X Terhadap Kebijakan Presiden Republik Indonesia Prabowo Subianto,” JURNAL FASILKOM, vol. 15, no. 3, pp. 443–455, 2025, doi: 10.37859/jf.v15i3.10385.
[29] K. Munawaroh and Alamsyah, “Performance Comparison of SVM, Naïve Bayes, and KNN Algorithms for Analysis of Public Opinion Sentiment Against COVID-19 Vaccination on Twitter,” Journal of Advances in Information Systems and Technology, vol. 4, no. 2, pp. 113–125, 2023, doi: 10.15294/jaist.v4i2.59493.
[30] N. W. S. Saraswati, C. P. Yanti, I. D. M. K. Muku, and A. P. R. D. Dewa, “Evaluation Analysis of the Necessity of Stemming and Lemmatization in Text Classification,” MATRIK: Jurnal Manajemen, Teknik Informatika, dan Rekayasa Komputer, vol. 24, no. 2, pp. 321–332, 2025, doi: 10.30812/matrik.v24i2.4833.
[31] E. F. Baharsyah, Armanto, T. H. B. Aviani, and C. Wulandari, “Analisis Sentimen Pengguna Terhadap Aplikasi Belajar Online Ruang Guru Pada Ulasan Google Play Store Menggunakan Algoritma Naïve Bayes dan Support Vector Machine,” Innovative: Journal Of Social Science Research, vol. 4, no. 3, pp. 2965–2979, 2024, doi: 10.31004/innovative.v4i3.10769.
[32] K. Hossain, L. Li, U. Ghani, and M. S. Hossain, “Unveiling the Impact of Service Attributes and Review Scores on Sentiment: A Deep Learning and Feature Engineering Approach to UberEats Reviews,” International Journal of Engineering Business Management., vol. 17, pp. 1–14, 2025, doi: 10.1177/18479790251341980.
[33] S. A. H. Bahtiar, C. K. Dewa, and A. Luthfi, “Comparison of Naïve Bayes and Logistic Regression in Sentiment Analysis on Marketplace Reviews Using Rating-Based Labeling,” Journal of Information Systems and Informatics, vol. 5, no. 3, pp. 915–927, 2023, doi: 10.51519/journalisi.v5i3.539.
[34] D. Shalikha and Alamsyah, “Improved Accuracy of Naïve Bayes Algorithm and Support Vector Machine Using Particle Swarm Optimization for Menstrual Cup Sentiment Analysis on Twitter,” Journal of Advances in Information Systems and Technology, vol. 4, no. 2, pp. 139–148, 2023, doi: 10.15294/jaist.v4i2.59561.
[35] M. Sivakumar, S. Parthasarathy, and T. Padmapriya, “Trade-off between Training and Testing Ratio in Machine Learning for Medical Image Processing,” PeerJ Computer Science, vol. 10, Art. no. e2245, 2024, doi: 10.7717/peerj-cs.2245.
[36] L. Zhang, “Features Extraction Based on Naive Bayes Algorithm and TF-IDF for News Classification,” PLoS One, vol. 20, no. 7, Art. no. e0327347, 2025, doi: 10.1371/journal.pone.0327347.
[37] N. A. Semary, W. Ahmed, K. Amin, P. Pławiak, and M. Hammad, “Enhancing Machine Learning-Based Sentiment Analysis Through Feature Extraction Techniques,” PLoS One, vol. 19, no. 2, Art. no. e0294968, 2024, doi: 10.1371/journal.pone.0294968.
[38] R. Obiedat et al., “Sentiment Analysis of Customers’ Reviews Using a Hybrid Evolutionary SVM-Based Approach in an Imbalanced Data Distribution,” IEEE Access, vol. 10, pp. 22260–22273, 2022, doi: 10.1109/ACCESS.2022.3149482.
[39] Y. Li, Y. Yang, P. Song, L. Duan, and R. Ren, “An Improved SMOTE Algorithm for Enhanced Imbalanced Data Classification by Expanding Sample Generation Space,” Scientific Reports, vol. 15, no. 1, Art. no. 23521, 2025, doi: 10.1038/s41598-025-09506-w.
[40] H. A. Salman, A. Kalakech, and A. Steiti, “Random Forest Algorithm Overview,” Babylonian Journal of Machine Learning, vol. 2024, pp. 69–79, 2024, doi: 10.58496/BJML/2024/007.
[41] M. . A. Azkiya and Y. N. Ifriza, “Analysis of Coding Stress Impact on Students Programming Skills with Random Forest and C4.5 Algorithms,” Journal of Computer Science and Information, vol. 18, no. 2, pp. 167–180, 2025, doi: 10.21609/jiki.v18i2.1487.
[42] J. Hu and S. Szymczak, “A Review on Longitudinal Data Analysis with Random Forest,” Briefings in Bioinformatics, vol. 24, no. 2, Art. no. bbad002, 2023, doi: 10.1093/bib/bbad002.
[43] S. Lee, C. Lee, K. G. I. Mun, and D. Kim, “Decision Tree Algorithm Considering Distances Between Classes,” IEEE Access, vol. 10, pp. 69750–69756, 2022, doi: 10.1109/ACCESS.2022.3187172.
[44] D. Valkenborg, A. Rousseau, M. Geubbelmans, and T. Burzykowski, “Support Vector Machines,” American Journal of Orthodontics and Dentofacial Orthopedics, vol. 164, no. 5, pp. 754–757, 2023, doi: 10.1016/j.ajodo.2023.08.003.
[45] O. I. Gifari, M. Adha, I. R. Hendrawan, and F. F. S. Durrand, “Analisis Sentimen Review Film Menggunakan TF-IDF dan Support Vector Machine,” JIFOTECH (Journal of Information Technology), vol. 2, no. 1, pp. 36–40, 2022, doi: 10.46229/jifotech.v2i1.330.
[46] A. Safitri, M. Risaldi, M. N. R. Alwi, D. F. Surianto, N. Fadilah, and J. M. Parenreng, “Improving Sentiment Classification of Kredit Pintar Reviews Using IndoBERT, SMOTE, and Stacking Ensemble,” Jurnal Teknik Informatika (JUTIF), vol. 7, no. 3, pp. 2411–2424, 2026, doi: 10.52436/1.jutif.2026.7.3.5342.
[47] S. Sathyanarayanan and B. R. Tantri, “Confusion Matrix-Based Performance Evaluation Metrics,” African Journal of Biomedical Research, vol. 27, no. 4S, pp. 4023–4031, 2024, doi: 10.53555/AJBR.v27i4S.4345.
[48] S. Pambudi, P. Setiaji, and W. A. Triyanto, “Sentiment Analysis of Fizzo Novel Application Using Support Vector Machine and Naïve Bayes Algorithm with SEMMA Framework,” Jurnal Teknik Informatika (JUTIF), vol. 6, no. 4, pp. 1861–1880, 2025, doi: 10.52436/1.jutif.2025.6.4.4875.
